Welcome to WebmasterWorld Guest from 54.146.59.202

Message Too Old, No Replies

Is Googlebot Eating my Bandwidth

Over 600mb on 2 very minor sites in 2 days

     
8:55 am on Feb 9, 2005 (gmt 0)

10+ Year Member



My bandwidth on a couple of sites is getting absolutely caned at the moment. My bandwidth usage is up 3 times on last month already! I've had to suspend one virtually unused site before it uses my entire reseller quota.

When I check the logs Googlebot/2.1 is at the top of the user agents list.
However, this much bandwidth seems excessive for crawling.

Any ideas?

9:13 am on Feb 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



In general, I NEVER limit bandwidth. If you're not 100% certain about who your closing the door on, don't close it.

600mb is not a lot of traffic - surely this wasn't even close to your reseller quota.

I'd unban everything and just let them feed - let them suck down as much as they can take if you'll benefit - bandwidth is relatively cheap.

1:59 pm on Feb 9, 2005 (gmt 0)

10+ Year Member



Actually, it seems to be eating up bandwidth faster than I said. I think I have 36gb - at the rate it's going I reckon it might get through a fair proportion of that.
The culprits are:
66.249.65.101, 66.249.65.67 and 66.249.65.68
2:03 pm on Feb 9, 2005 (gmt 0)

10+ Year Member



This is one of those good problems to have: a gold-laden stream is heading toward my property - what should I do?
2:09 pm on Feb 9, 2005 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Are you sure Googlebot is not caught in a spider trap? Common culprits include session IDs or scripts such as calendars which have an infinite number of pages (eg. there's always a link to the next and previous month, to infinity).

If it is becoming a problem, report it to googlebot@google.com, and if necessary add some robots meta tags to exclude the bot from areas where it has become trapped.

  • Googlebot guide [google.com]
  •  

    Featured Threads

    Hot Threads This Week

    Hot Threads This Month