homepage Welcome to WebmasterWorld Guest from 54.197.171.109
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque & physics

Webmaster General Forum

    
Googlebot hits me.
Why googlebot is so agressive
qmaster




msg:3303659
 8:36 pm on Apr 5, 2007 (gmt 0)

Hi,

Just look to these numbers, it's crazy.
Google crawler is 100 times more aggressive than Yahoo's crawler.
How do I reduce the frequency? Also take a lot of bandwidth. This is this month only! 5 Days. Also the reach from google to my site is only double than the one from yahoo...

Googlebot 14268+191 223.37 MB
Yahoo Slurp 144+6 31.30 MB
MSNBot 25+30 839.49 KB

This are stats generate by AWSTATS

Any ideas how to I reduce the frequency?

 

londrum




msg:3303671
 8:45 pm on Apr 5, 2007 (gmt 0)

you can slow it down in google sitemaps.
open up an account, and there is a setting to reduce it to a slow crawl.
and if you do it like that, then it won't affect any of the other search engine spiders that are crawling you okay.

monkeythumpa




msg:3303805
 11:02 pm on Apr 5, 2007 (gmt 0)

That is not that much. Are you seeing long site response times when they are crawling your site? Unless your site is hammered so badly that it is getting slow, don't worry about it. More spidering is usually a good thing, especially for Yahoo and MSN where it can take ages between get spidered and getting indexed.

rocknbil




msg:3303962
 4:20 am on Apr 6, 2007 (gmt 0)

You can also reduce the crawl rate in google's webmaster tools control panel.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved