Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: phranque

Message Too Old, No Replies

How to tell Inktomi and other good spiders to slow down

     
7:01 pm on Dec 13, 2006 (gmt 0)

New User

10+ Year Member

joined:Jan 31, 2006
posts:24
votes: 0


Every once in a while, my discussion board will get utterly bombarded by "good" search engine spiders, especially Inktomi.

Is there a technical way to respond to the spider: "Hey, I'm busy right now; please come back later, and by the way, don't penalize me for responding in this way". (Note that the discussion board provides for me to determine the current user load that I would use in determining when to respond in this manner.)

Hits to a discussion board suck up database resources, even with SQL caching built in, so I need to be able to tell the spiders to not bother the site for a while.

Any ideas?

12:32 am on Dec 14, 2006 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


you can try the following in the robots.txt file:
Crawl-delay: xx
where xx is seconds delay between crawls.
not all spiders support this.
[support.inktomi.com...]