Page is a not externally linkable
jrobbio - 1:12 am on May 18, 2003 (gmt 0)
With sites expanding and changing it should be said that there will be many a webmaster not too happy to find unwanted material appearing in the SERPS and striving to find the removal form or ban the bot altogether. So my question goes out to you is:
With the likes of Freshbot and Grub making updating of sites faster and faster, there is obviously the possibility that the robot will overstep the mark and inadvertantly crawl banned material.
How often (in hours, days, bursts or whatever measurement) do you feel it would be appropriate for the robots.txt to be checked?