Welcome to WebmasterWorld Guest from 34.229.24.100

Forum Moderators: Ocean10000

Message Too Old, No Replies

How Often To Check Robots.txt?

     
5:53 pm on Jun 1, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 11, 2004
posts: 62
votes: 0


Hi All,

Last summer I got beat up a bit (and rightly so) in this thread because my robot wasn't yet checking robots.txt file.

[webmasterworld.com...]

I shelved the project for a bit, it's back and I now have my crawler first checking robots.txt for permission.

I want to make robots.txt visits separate from the crawl. What's the norm for how long I can appropriately cache the permissions found in a site's robots.txt?

thanks - jeff
7:28 pm on June 1, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Less than 24 hours would be ideal.