Forum Moderators: open
Is there anything to put in robots.txt to slow the bot down?
Thanks,
Chris
You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server.
Setting a crawl-delay of 20 seconds for Yahoo! Slurp would look something like:
User-agent: Slurp
Crawl-delay: 20
[help.yahoo.com...]