Welcome to WebmasterWorld Guest from 54.234.114.182

Forum Moderators: open

Message Too Old, No Replies

12 days between fetch of robots.txt and crawl?

     
9:01 pm on Jun 28, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 20, 2002
posts:194
votes: 0


Ask Jeeves fetched "robots.txt" on Jun, 16th and then fetched some pages on Jun, 28th. Is this really considered reasonable? I would expect them to get the "robots.txt" on the same day they actually crawl the page.

There are some (stupid) robots that get robots.txt, page, robots.txt, next page, ... I don't like that either but 12 days? Can anyone confirm?

10:35 am on July 11, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 20, 2002
posts:194
votes: 0


They got robots.txt again on July, 8th. So do they only fetch it once a month and compare before updates?
11:37 am on July 13, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 21, 2001
posts:2489
votes: 0


Luma,

maybe. is it different robots that are crawling the different types of pages, this is not unusual, then a big cross-correlation, or weighting process takes place once all the pieces are in.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members