Using meta tags to slow Googlebot down.

bandwidth taking a hit!

[esllou]

I have a problem with Googlebot, but a nice one. I have 13,000 recently created product pages which google recently added to its index. Took quite a hit on the bandwidth getting them all indexed. The problem is that now, Googlebot keeps returning and re-indexing them and my bandwidth is continuing to get hit! I clearly don't want to use robots.txt to stop this as that is too much of a sledgehammer to use....can I use robots meta tags?

I have never used robots meta tags and heard Googlebot didn't obey them anyway. I thought there was a tag such as NOINDEX 7 DAYS or something similar I could use to slow bots down.

Would anyone else have a better idea or know if this would work with Googlebot?

Appreciated.

[pontifex]

If you are using Linux, you may want to try the command "iptables"...

we had an ugly attack of altavista, taking 20 pages a second and the server just went nuts..

If you have "iptables" installed on your Linux box, create a script with any editor:

#!/bin/sh

IFACE="eth0"

iptables -I INPUT -p tcp --dport 80 -s 216.39.48.0/20 -m limit --limit 180/min -j ACCEPT
iptables -I INPUT 2 -p tcp --dport 80 -s 216.39.48.0/20 -j DROP

exchange the IPs of AltaVista with the Google network you want to slow down...

so that network would be slowed to 180 pages per minute... you might play around with that one.
P!

[BlueSky]

Sounds like he wants his pages indexed less frequently not that Googlebot is going too fast.

There's a revisit metatag but I don't know if Googlebot honors it or not. You can try it out.

<meta name="Revisit-After" content="7 Days">

[ThomasB]

Maybe an additional header could solve the problem:
[webmasterworld.com...]

[nakulgoyal]

GoogleBOT hits over 4GB bandwidth on one of my clients website. :-)