homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

12 days between fetch of robots.txt and crawl?

10+ Year Member

Msg#: 142 posted 9:01 pm on Jun 28, 2002 (gmt 0)

Ask Jeeves fetched "robots.txt" on Jun, 16th and then fetched some pages on Jun, 28th. Is this really considered reasonable? I would expect them to get the "robots.txt" on the same day they actually crawl the page.

There are some (stupid) robots that get robots.txt, page, robots.txt, next page, ... I don't like that either but 12 days? Can anyone confirm?



10+ Year Member

Msg#: 142 posted 10:35 am on Jul 11, 2002 (gmt 0)

They got robots.txt again on July, 8th. So do they only fetch it once a month and compare before updates?


WebmasterWorld Senior Member 10+ Year Member

Msg#: 142 posted 11:37 am on Jul 13, 2002 (gmt 0)


maybe. is it different robots that are crawling the different types of pages, this is not unusual, then a big cross-correlation, or weighting process takes place once all the pieces are in.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved