Forum Moderators: open

Message Too Old, No Replies

Any way to slow Yahoo's bot down?

It is really hammering the site of a client of mine.

         

onlineshrine

3:37 pm on Jun 29, 2004 (gmt 0)

10+ Year Member



A client at the managed dedicated server hosting company I work for is having his server hammered by Yahoo's bot. I was wondering if there was something I could put in robots.txt that would slow the crawler down. He still wants it to index his site, just not with as much fervor as it currently does.

Is there anything to put in robots.txt to slow the bot down?

Thanks,
Chris

Tim

8:28 pm on Jun 29, 2004 (gmt 0)

10+ Year Member



How can I reduce the number of requests you make on my web site?
Since we crawl billions of pages from the entire Web, we use a large number of systems for web crawling. Therefore your web server may log requests from a number of different Yahoo! crawler client IP addresses. The different crawler systems are coordinated to limit the activity on any single web server. We determine a single "web server" by IP address, so if your host is serving multiple IPs it may see higher levels of activity.
There is a Yahoo! Slurp-specific extension to robots.txt which allows you to set a lower limit on our crawler request rate.

You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server.

Setting a crawl-delay of 20 seconds for Yahoo! Slurp would look something like:

User-agent: Slurp
Crawl-delay: 20

[help.yahoo.com...]

Katz

9:16 am on Jun 30, 2004 (gmt 0)

10+ Year Member



Tim,

If I change the ip of a domain what is the fastest way to make Slurp crawl the new ip and update its dbase with the new ip corresponding to this domain.