homepage Welcome to WebmasterWorld Guest from 54.226.173.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Google Deep Crawling My site Today.
Chief




msg:110207
 3:30 pm on May 1, 2003 (gmt 0)

Crawling again. About 20+ bots on my site. Crawl1....(but is the crawl or crawler the deepbot? I forget.

Does this mean the update is farther away than we would like?

 

John_Caius




msg:110208
 3:34 pm on May 1, 2003 (gmt 0)

216 is the deepbot, 64 is the freshbot.

PeterHo




msg:110209
 2:16 pm on May 2, 2003 (gmt 0)

How do you know that bot's have visited your page?
By the IP-numbers of you visitors?

ncsuk




msg:110210
 2:22 pm on May 2, 2003 (gmt 0)

My site has been deep crawled as well and some pages 4 levels deep have been picked up

speda1




msg:110211
 5:42 pm on May 2, 2003 (gmt 0)

You cannot tell if its deepbot by the pages that are grabbed. You have to look at the IP of the bot. See John's post above.

tomapple




msg:110212
 6:31 pm on May 2, 2003 (gmt 0)

havn't seen the deepbot today, just "freshie" hitting a bunch of pages

taxpod




msg:110213
 6:31 pm on May 2, 2003 (gmt 0)

I'm seeing only freshy.

kmtell




msg:110214
 8:22 pm on May 2, 2003 (gmt 0)

I have a ton of 216.* hits on my site today.

speda1




msg:110215
 8:59 pm on May 2, 2003 (gmt 0)

With a user agent of 'Googlebot'?

markusf




msg:110216
 9:11 pm on May 2, 2003 (gmt 0)

I get a lot of these on my site..

216.239.33.5
Nokia6610/1.0 (4.18) Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)

speda1




msg:110217
 1:27 am on May 3, 2003 (gmt 0)

That looks human to me.

matrix_neo




msg:110218
 4:18 am on May 3, 2003 (gmt 0)

What is the difference between up date and google dance (I believe both are the same). Do you guys mean update is PR calculation after the google Or PR calculation is also happening simultaneously?

speda1




msg:110219
 5:24 pm on May 3, 2003 (gmt 0)

The update and the dance are the same thing. Each of the google data centers are updated with the new index. Because Google load balances with DNS, and because it takes some time to update all of the data centers, search results fluctuate during this time, hence the 'dance'.

PR is most likely calculated during the period between the deep crawl and the start of the dance. We are in this period now.

Jesse_Smith




msg:110220
 11:54 pm on May 3, 2003 (gmt 0)

freshBot: 64.68.82.* bah, listed for only a few days, short dinner date then it dumps you.
deepcrawler: 216.239.46.* Good bot, it likes you, listed until death do you depart. So don't make him mad or it will divorce your site. You can divorce the Googlebot by using your robot.txt file. It's much cheaper and faster than going to the courts.

If it's the Googlebot it will say

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved