homepage Welcome to WebmasterWorld Guest from 54.163.89.8
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
googlebot consume heavy bandwith
br4inwash3r

5+ Year Member



 
Msg#: 4556 posted 10:01 am on Apr 19, 2006 (gmt 0)

one of my site exceeds it's bandwith usage limit, yet it's not even near the end of the month.

before the limit's reached, i've noticed a suspicious increase of bandwith usage. the CPanel interface says we have used over than 5 gigs just for this month! then i check the out stats to look for a suspect, and end up finding out that for the past few days there have been a heavy activity from googlebot. about 300mb per visits!

this site is a phpbb forum. i know about the trouble that the bot can run when crawling forums, but is this normal? does anyone have similar experience?

well, if so. where can i complain about this? the hosting company? any enlightment will be very appreciated.

geez man, getting out of bandwith this early really suxx!

 

larryhatch

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4556 posted 10:17 am on Apr 19, 2006 (gmt 0)

My limited experience is that its much cheaper to pay a few bucks more for higher bandwidth
than to pay overages after the fact. Both are better than simply getting shut down of course.

5000MB/300MB means Google can wipe you out in 16 or 17 crawls.
They must be deep crawling, something a lot of people here would welcome.
I would just pay the extra pennies for higher bandwidth service. -Larry

br4inwash3r

5+ Year Member



 
Msg#: 4556 posted 7:27 pm on Apr 19, 2006 (gmt 0)

thx for the reply. i've just asked the hosting company wether we can buy more bandwith for the time being. no replies yet...

but we're already planning on moving though. the service sucks. and yes, the next hosting company will have a lot bigger bandwith.

anyway, on an average, how much bandwith does googlebot takes when crawling forums anyway?

thx again.

Mistra

5+ Year Member



 
Msg#: 4556 posted 3:04 am on May 6, 2006 (gmt 0)

It is time for you to implement Google Sitemap. Good luck with your new host.

carguy84

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4556 posted 8:00 am on May 7, 2006 (gmt 0)

what is the googlebot hitting that is causing so much bandwidth usage? It can't be all HTML, right? If it's hitting images, just block off your images folder in the robots.txt file and the bots won't pull down all your images.

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4556 posted 8:51 am on May 7, 2006 (gmt 0)

this site is a phpbb forum. i know about the trouble that the bot can run when crawling forums, but is this normal?

Sounds like you're serving session ID's up with your URL's.

That would be the first thing to check.

TJ

arran

5+ Year Member



 
Msg#: 4556 posted 9:14 am on May 7, 2006 (gmt 0)

What is the googlebot hitting that is causing so much bandwidth usage? It can't be all HTML, right? If it's hitting images, just block off your images folder in the robots.txt file and the bots won't pull down all your images.

Images are spidered by Google Imagebot not Googlebot.

As TJ said, it sounds like a session id issue.

wmuser

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4556 posted 10:17 pm on May 7, 2006 (gmt 0)

I dont know your website size/number of pages however Googlebot should consume that much bandwidth

br4inwash3r

5+ Year Member



 
Msg#: 4556 posted 8:20 am on May 14, 2006 (gmt 0)

TJ: what can I do to those session ID?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved