Welcome to WebmasterWorld Guest from

Forum Moderators: phranque

Message Too Old, No Replies

Googlebot hits me.

Why googlebot is so agressive

8:36 pm on Apr 5, 2007 (gmt 0)

New User

10+ Year Member

joined:Apr 5, 2007
votes: 0


Just look to these numbers, it's crazy.
Google crawler is 100 times more aggressive than Yahoo's crawler.
How do I reduce the frequency? Also take a lot of bandwidth. This is this month only! 5 Days. Also the reach from google to my site is only double than the one from yahoo...

Googlebot 14268+191 223.37 MB
Yahoo Slurp 144+6 31.30 MB
MSNBot 25+30 839.49 KB

This are stats generate by AWSTATS

Any ideas how to I reduce the frequency?

8:45 pm on Apr 5, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Feb 12, 2006
votes: 93

you can slow it down in google sitemaps.
open up an account, and there is a setting to reduce it to a slow crawl.
and if you do it like that, then it won't affect any of the other search engine spiders that are crawling you okay.
11:02 pm on Apr 5, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 2, 2004
votes: 0

That is not that much. Are you seeing long site response times when they are crawling your site? Unless your site is hammered so badly that it is getting slow, don't worry about it. More spidering is usually a good thing, especially for Yahoo and MSN where it can take ages between get spidered and getting indexed.
4:20 am on Apr 6, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member rocknbil is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 28, 2004
votes: 0

You can also reduce the crawl rate in google's webmaster tools control panel.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members