Forum Moderators: open

Message Too Old, No Replies

how can I slow down googlebot?

slow down googlebot spider queries

         

denby

12:47 pm on Nov 28, 2004 (gmt 0)

10+ Year Member



Is there an equivalent robots.txt entry for googlebot to the one for slurp that says:

Crawl-delay: xx

where xx is the delay in seconds between spider queries on my site pages?

Or some other workaround which would suit a php script and another for a perl script?

Denby

Vimes

4:26 am on Nov 29, 2004 (gmt 0)

10+ Year Member



I’m no expert on this but as I understand it, if the bot is taxing your server too much then the bot will automatically back off and not hit as hard.

I don't think Googlebot takes any notice of revisit meta tags.

But i could be wrong on that.

Vimes.

walkman

4:37 am on Nov 29, 2004 (gmt 0)



Google wants to know when their bot goes wild. Search their webmaster FAQS and email them with the info.