homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Google bot going mad on my site, 10 pages/sec

 9:49 pm on Jan 23, 2011 (gmt 0)

lol since i opened a new section on my website google bot is requesting like 10pages every second all day long.

Funny thing is he is crawling teh same page even if they handle if-modified-since

lucky i have disabled the apache logger... 10gb of logs aren't fun to manage

add: just want to share with you this behavoiour.
add2: bot' ip seem to be only 2 (starting with 66.249)



 11:28 pm on Jan 23, 2011 (gmt 0)

That's not normal googlebot behavior. Have you tested it with this approach: How to Verify Googlebot [webmasterworld.com]?


 11:56 pm on Jan 23, 2011 (gmt 0)

dns and ip for both match. (it's googlebot)

edit: why do u think it's not normal? (for large website this can only speed up the indexing process... the only problem is it's requesting pages that has already indexed like yesterday)

[edited by: xnavigator at 12:09 am (utc) on Jan 24, 2011]


 12:07 am on Jan 24, 2011 (gmt 0)

Ah, I misunderstood. I thought you were getting repeated requests within a second for the same URL - even after serving a 304 response. Well, repeatedly checking on a 304 response doesn't take much time at all - as long as other URLs are also getting requested.


 12:10 am on Jan 24, 2011 (gmt 0)

yes sorry my bad..

not only 1 url... but every between the ~10000 possibile urls


 6:46 am on Jan 24, 2011 (gmt 0)

We get about 10 requests a second per domain they crawl. In the webmaster tool that seems to be the highest setting we can set if we set it manually...


 11:53 am on Jan 24, 2011 (gmt 0)

in my webmaster tools i can set max only 2.2 requests per second.. (while currenty it's crawling 10 per sec)

But anyway I never believed webmaster tool really influenced google :D


 12:12 pm on Jan 24, 2011 (gmt 0)

I am happy to know that webmaster is helping in crawling the pages so faster.


 1:53 pm on Jan 24, 2011 (gmt 0)

Have you started an SEM campaign against this new section of the site? I have seen googlbot go crazy for SEM against large sites. It seems that it is set to do one crawl per keyword that you bid on every day. I've seen it crawl the same landing page that has 100 keywords bid against it 100 times a day. When you are bidding on millions of keywords, the crawl rates can be debilitating.


 2:03 pm on Jan 24, 2011 (gmt 0)

no i just linked them from other pages

Robert Charlton

 1:21 am on Jan 25, 2011 (gmt 0)

no i just linked them from other pages

That explains a lot. They would of course follow the new links.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved