homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Is Googlebot Eating my Bandwidth
Over 600mb on 2 very minor sites in 2 days
surfgatinho

10+ Year Member



 
Msg#: 27935 posted 8:55 am on Feb 9, 2005 (gmt 0)

My bandwidth on a couple of sites is getting absolutely caned at the moment. My bandwidth usage is up 3 times on last month already! I've had to suspend one virtually unused site before it uses my entire reseller quota.

When I check the logs Googlebot/2.1 is at the top of the user agents list.
However, this much bandwidth seems excessive for crawling.

Any ideas?

 

Chico_Loco

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 27935 posted 9:13 am on Feb 9, 2005 (gmt 0)

In general, I NEVER limit bandwidth. If you're not 100% certain about who your closing the door on, don't close it.

600mb is not a lot of traffic - surely this wasn't even close to your reseller quota.

I'd unban everything and just let them feed - let them suck down as much as they can take if you'll benefit - bandwidth is relatively cheap.

surfgatinho

10+ Year Member



 
Msg#: 27935 posted 1:59 pm on Feb 9, 2005 (gmt 0)

Actually, it seems to be eating up bandwidth faster than I said. I think I have 36gb - at the rate it's going I reckon it might get through a fair proportion of that.
The culprits are:
66.249.65.101, 66.249.65.67 and 66.249.65.68

Galtego

10+ Year Member



 
Msg#: 27935 posted 2:03 pm on Feb 9, 2005 (gmt 0)

This is one of those good problems to have: a gold-laden stream is heading toward my property - what should I do?

encyclo

WebmasterWorld Senior Member encyclo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 27935 posted 2:09 pm on Feb 9, 2005 (gmt 0)

Are you sure Googlebot is not caught in a spider trap? Common culprits include session IDs or scripts such as calendars which have an infinite number of pages (eg. there's always a link to the next and previous month, to infinity).

If it is becoming a problem, report it to googlebot@google.com, and if necessary add some robots meta tags to exclude the bot from areas where it has become trapped.

  • Googlebot guide [google.com]
  • Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Google / Google SEO News and Discussion
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved