Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Question about specific deepcrawling bots used by Google

It may be a silly question but I can't find anything definitive on this

6:32 am on Dec 27, 2002 (gmt 0)

New User

10+ Year Member

joined:Nov 26, 2002
votes: 0

Is it possible to be deep crawled by a 64.68.82.xx bot or is deep crawling absolutely, *only* done by bots with the 216 prefix?
9:26 am on Dec 27, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Nov 30, 2002
votes: 0

according to all the discussion here the 64 is the fresh bot and for deep crawl it is done only by the 216
10:53 pm on Dec 27, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 2, 2002
votes: 0

However, 64.etc will sometimes visit root directory pages on some sites in addition to the index page - I think independently, not from following a link off the index page - but it won't follow links.
10:59 pm on Dec 27, 2002 (gmt 0)

Preferred Member

10+ Year Member

joined:May 1, 2002
votes: 0

Freshbot can index thousands of pages and seem just like the deep crawl. So it must follow links. I have also seen new sites get hits from the freshbot off of links.
12:29 am on Dec 28, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
votes: 0

There is nothing definitive. Just because google "tends" to do things one way or the other, they are not limited to stick with the rules that members her try to put on them.

In either October or November, one of my pages was hit by the freshbot, and only the freshbot, during the main crawl time frame. There was no way for this page to achieve fresh status, and it appeared in the update, so the freshbot must have been acting as a deep crawler at that point. That same month, a couple of my normal "fresh" pages were hit by the deep crawler late in the month. As those pages always appear fresh, it is hard to say that the deep crawler was acting as a freshbot, but I also cannot say that it wasn't, because it sure was acting like one.

In December, I have not noticed any bots behaving differently, but that doesn't mean that they can't change their functionality.