homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Question about specific deepcrawling bots used by Google
It may be a silly question but I can't find anything definitive on this

10+ Year Member

Msg#: 7947 posted 6:32 am on Dec 27, 2002 (gmt 0)

Is it possible to be deep crawled by a 64.68.82.xx bot or is deep crawling absolutely, *only* done by bots with the 216 prefix?



10+ Year Member

Msg#: 7947 posted 9:26 am on Dec 27, 2002 (gmt 0)

according to all the discussion here the 64 is the fresh bot and for deep crawl it is done only by the 216


10+ Year Member

Msg#: 7947 posted 10:53 pm on Dec 27, 2002 (gmt 0)

However, 64.etc will sometimes visit root directory pages on some sites in addition to the index page - I think independently, not from following a link off the index page - but it won't follow links.


10+ Year Member

Msg#: 7947 posted 10:59 pm on Dec 27, 2002 (gmt 0)

Freshbot can index thousands of pages and seem just like the deep crawl. So it must follow links. I have also seen new sites get hits from the freshbot off of links.


WebmasterWorld Senior Member bigdave us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 7947 posted 12:29 am on Dec 28, 2002 (gmt 0)

There is nothing definitive. Just because google "tends" to do things one way or the other, they are not limited to stick with the rules that members her try to put on them.

In either October or November, one of my pages was hit by the freshbot, and only the freshbot, during the main crawl time frame. There was no way for this page to achieve fresh status, and it appeared in the update, so the freshbot must have been acting as a deep crawler at that point. That same month, a couple of my normal "fresh" pages were hit by the deep crawler late in the month. As those pages always appear fresh, it is hard to say that the deep crawler was acting as a freshbot, but I also cannot say that it wasn't, because it sure was acting like one.

In December, I have not noticed any bots behaving differently, but that doesn't mean that they can't change their functionality.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved