|Question about specific deepcrawling bots used by Google|
It may be a silly question but I can't find anything definitive on this
| 6:32 am on Dec 27, 2002 (gmt 0)|
Is it possible to be deep crawled by a 64.68.82.xx bot or is deep crawling absolutely, *only* done by bots with the 216 prefix?
| 9:26 am on Dec 27, 2002 (gmt 0)|
according to all the discussion here the 64 is the fresh bot and for deep crawl it is done only by the 216
| 10:53 pm on Dec 27, 2002 (gmt 0)|
However, 64.etc will sometimes visit root directory pages on some sites in addition to the index page - I think independently, not from following a link off the index page - but it won't follow links.
| 10:59 pm on Dec 27, 2002 (gmt 0)|
Freshbot can index thousands of pages and seem just like the deep crawl. So it must follow links. I have also seen new sites get hits from the freshbot off of links.
| 12:29 am on Dec 28, 2002 (gmt 0)|
There is nothing definitive. Just because google "tends" to do things one way or the other, they are not limited to stick with the rules that members her try to put on them.
In either October or November, one of my pages was hit by the freshbot, and only the freshbot, during the main crawl time frame. There was no way for this page to achieve fresh status, and it appeared in the update, so the freshbot must have been acting as a deep crawler at that point. That same month, a couple of my normal "fresh" pages were hit by the deep crawler late in the month. As those pages always appear fresh, it is hard to say that the deep crawler was acting as a freshbot, but I also cannot say that it wasn't, because it sure was acting like one.
In December, I have not noticed any bots behaving differently, but that doesn't mean that they can't change their functionality.