Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Is Googlebot Eating my Bandwidth

Over 600mb on 2 very minor sites in 2 days



8:55 am on Feb 9, 2005 (gmt 0)

10+ Year Member

My bandwidth on a couple of sites is getting absolutely caned at the moment. My bandwidth usage is up 3 times on last month already! I've had to suspend one virtually unused site before it uses my entire reseller quota.

When I check the logs Googlebot/2.1 is at the top of the user agents list.
However, this much bandwidth seems excessive for crawling.

Any ideas?


9:13 am on Feb 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

In general, I NEVER limit bandwidth. If you're not 100% certain about who your closing the door on, don't close it.

600mb is not a lot of traffic - surely this wasn't even close to your reseller quota.

I'd unban everything and just let them feed - let them suck down as much as they can take if you'll benefit - bandwidth is relatively cheap.


1:59 pm on Feb 9, 2005 (gmt 0)

10+ Year Member

Actually, it seems to be eating up bandwidth faster than I said. I think I have 36gb - at the rate it's going I reckon it might get through a fair proportion of that.
The culprits are:, and


2:03 pm on Feb 9, 2005 (gmt 0)

10+ Year Member

This is one of those good problems to have: a gold-laden stream is heading toward my property - what should I do?


2:09 pm on Feb 9, 2005 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Are you sure Googlebot is not caught in a spider trap? Common culprits include session IDs or scripts such as calendars which have an infinite number of pages (eg. there's always a link to the next and previous month, to infinity).

If it is becoming a problem, report it to googlebot@google.com, and if necessary add some robots meta tags to exclude the bot from areas where it has become trapped.

  • Googlebot guide [google.com]

    Featured Threads

    Hot Threads This Week

    Hot Threads This Month