Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000

Message Too Old, No Replies

How Often To Check Robots.txt?

5:53 pm on Jun 1, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 11, 2004
posts: 62
votes: 0

Hi All,

Last summer I got beat up a bit (and rightly so) in this thread because my robot wasn't yet checking robots.txt file.


I shelved the project for a bit, it's back and I now have my crawler first checking robots.txt for permission.

I want to make robots.txt visits separate from the crawl. What's the norm for how long I can appropriately cache the permissions found in a site's robots.txt?

thanks - jeff
7:28 pm on June 1, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
votes: 0

Less than 24 hours would be ideal.