Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- How often should a robot check the robots.txt file?


jrobbio - 1:12 am on May 18, 2003 (gmt 0)


With the likes of Freshbot and Grub making updating of sites faster and faster, there is obviously the possibility that the robot will overstep the mark and inadvertantly crawl banned material.

With sites expanding and changing it should be said that there will be many a webmaster not too happy to find unwanted material appearing in the SERPS and striving to find the removal form or ban the bot altogether.

So my question goes out to you is:
How often (in hours, days, bursts or whatever measurement) do you feel it would be appropriate for the robots.txt to be checked?


Thread source:: http://www.webmasterworld.com/robots_txt/61.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com