homepage Welcome to WebmasterWorld Guest from 54.161.228.29
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Question about specific deepcrawling bots used by Google
It may be a silly question but I can't find anything definitive on this
NicoleA




msg:48597
 6:32 am on Dec 27, 2002 (gmt 0)

Is it possible to be deep crawled by a 64.68.82.xx bot or is deep crawling absolutely, *only* done by bots with the 216 prefix?

 

jamesyap




msg:48598
 9:26 am on Dec 27, 2002 (gmt 0)

according to all the discussion here the 64 is the fresh bot and for deep crawl it is done only by the 216

bobmark




msg:48599
 10:53 pm on Dec 27, 2002 (gmt 0)

However, 64.etc will sometimes visit root directory pages on some sites in addition to the index page - I think independently, not from following a link off the index page - but it won't follow links.

taxpod




msg:48600
 10:59 pm on Dec 27, 2002 (gmt 0)

Freshbot can index thousands of pages and seem just like the deep crawl. So it must follow links. I have also seen new sites get hits from the freshbot off of links.

BigDave




msg:48601
 12:29 am on Dec 28, 2002 (gmt 0)

There is nothing definitive. Just because google "tends" to do things one way or the other, they are not limited to stick with the rules that members her try to put on them.

In either October or November, one of my pages was hit by the freshbot, and only the freshbot, during the main crawl time frame. There was no way for this page to achieve fresh status, and it appeared in the update, so the freshbot must have been acting as a deep crawler at that point. That same month, a couple of my normal "fresh" pages were hit by the deep crawler late in the month. As those pages always appear fresh, it is hard to say that the deep crawler was acting as a freshbot, but I also cannot say that it wasn't, because it sure was acting like one.

In December, I have not noticed any bots behaving differently, but that doesn't mean that they can't change their functionality.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved