Welcome to WebmasterWorld Guest from 50.16.112.199

Forum Moderators: open

Question about specific deepcrawling bots used by Google

It may be a silly question but I can't find anything definitive on this

   
6:32 am on Dec 27, 2002 (gmt 0)

10+ Year Member



Is it possible to be deep crawled by a 64.68.82.xx bot or is deep crawling absolutely, *only* done by bots with the 216 prefix?
9:26 am on Dec 27, 2002 (gmt 0)

10+ Year Member



according to all the discussion here the 64 is the fresh bot and for deep crawl it is done only by the 216
10:53 pm on Dec 27, 2002 (gmt 0)

10+ Year Member



However, 64.etc will sometimes visit root directory pages on some sites in addition to the index page - I think independently, not from following a link off the index page - but it won't follow links.
10:59 pm on Dec 27, 2002 (gmt 0)

10+ Year Member



Freshbot can index thousands of pages and seem just like the deep crawl. So it must follow links. I have also seen new sites get hits from the freshbot off of links.
12:29 am on Dec 28, 2002 (gmt 0)

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member



There is nothing definitive. Just because google "tends" to do things one way or the other, they are not limited to stick with the rules that members her try to put on them.

In either October or November, one of my pages was hit by the freshbot, and only the freshbot, during the main crawl time frame. There was no way for this page to achieve fresh status, and it appeared in the update, so the freshbot must have been acting as a deep crawler at that point. That same month, a couple of my normal "fresh" pages were hit by the deep crawler late in the month. As those pages always appear fresh, it is hard to say that the deep crawler was acting as a freshbot, but I also cannot say that it wasn't, because it sure was acting like one.

In December, I have not noticed any bots behaving differently, but that doesn't mean that they can't change their functionality.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month