homepage Welcome to WebmasterWorld Guest from 23.20.34.25
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Is Googlebot Eating my Bandwidth
Over 600mb on 2 very minor sites in 2 days
surfgatinho




msg:739176
 8:55 am on Feb 9, 2005 (gmt 0)

My bandwidth on a couple of sites is getting absolutely caned at the moment. My bandwidth usage is up 3 times on last month already! I've had to suspend one virtually unused site before it uses my entire reseller quota.

When I check the logs Googlebot/2.1 is at the top of the user agents list.
However, this much bandwidth seems excessive for crawling.

Any ideas?

 

Chico_Loco




msg:739177
 9:13 am on Feb 9, 2005 (gmt 0)

In general, I NEVER limit bandwidth. If you're not 100% certain about who your closing the door on, don't close it.

600mb is not a lot of traffic - surely this wasn't even close to your reseller quota.

I'd unban everything and just let them feed - let them suck down as much as they can take if you'll benefit - bandwidth is relatively cheap.

surfgatinho




msg:739178
 1:59 pm on Feb 9, 2005 (gmt 0)

Actually, it seems to be eating up bandwidth faster than I said. I think I have 36gb - at the rate it's going I reckon it might get through a fair proportion of that.
The culprits are:
66.249.65.101, 66.249.65.67 and 66.249.65.68

Galtego




msg:739179
 2:03 pm on Feb 9, 2005 (gmt 0)

This is one of those good problems to have: a gold-laden stream is heading toward my property - what should I do?

encyclo




msg:739180
 2:09 pm on Feb 9, 2005 (gmt 0)

Are you sure Googlebot is not caught in a spider trap? Common culprits include session IDs or scripts such as calendars which have an infinite number of pages (eg. there's always a link to the next and previous month, to infinity).

If it is becoming a problem, report it to googlebot@google.com, and if necessary add some robots meta tags to exclude the bot from areas where it has become trapped.

  • Googlebot guide [google.com]
  • Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Google / Google SEO News and Discussion
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
    © Webmaster World 1996-2014 all rights reserved