homepage Welcome to WebmasterWorld Guest from 54.211.34.105
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google bot going mad on my site, 10 pages/sec
xnavigator

5+ Year Member



 
Msg#: 4257070 posted 9:49 pm on Jan 23, 2011 (gmt 0)

lol since i opened a new section on my website google bot is requesting like 10pages every second all day long.

Funny thing is he is crawling teh same page even if they handle if-modified-since

lucky i have disabled the apache logger... 10gb of logs aren't fun to manage

add: just want to share with you this behavoiour.
add2: bot' ip seem to be only 2 (starting with 66.249)

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4257070 posted 11:28 pm on Jan 23, 2011 (gmt 0)

That's not normal googlebot behavior. Have you tested it with this approach: How to Verify Googlebot [webmasterworld.com]?

xnavigator

5+ Year Member



 
Msg#: 4257070 posted 11:56 pm on Jan 23, 2011 (gmt 0)

dns and ip for both match. (it's googlebot)


edit: why do u think it's not normal? (for large website this can only speed up the indexing process... the only problem is it's requesting pages that has already indexed like yesterday)

[edited by: xnavigator at 12:09 am (utc) on Jan 24, 2011]

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4257070 posted 12:07 am on Jan 24, 2011 (gmt 0)

Ah, I misunderstood. I thought you were getting repeated requests within a second for the same URL - even after serving a 304 response. Well, repeatedly checking on a 304 response doesn't take much time at all - as long as other URLs are also getting requested.

xnavigator

5+ Year Member



 
Msg#: 4257070 posted 12:10 am on Jan 24, 2011 (gmt 0)

yes sorry my bad..

not only 1 url... but every between the ~10000 possibile urls

askbjoernhansen

5+ Year Member



 
Msg#: 4257070 posted 6:46 am on Jan 24, 2011 (gmt 0)

We get about 10 requests a second per domain they crawl. In the webmaster tool that seems to be the highest setting we can set if we set it manually...

xnavigator

5+ Year Member



 
Msg#: 4257070 posted 11:53 am on Jan 24, 2011 (gmt 0)

in my webmaster tools i can set max only 2.2 requests per second.. (while currenty it's crawling 10 per sec)

But anyway I never believed webmaster tool really influenced google :D

ChristeneMint



 
Msg#: 4257070 posted 12:12 pm on Jan 24, 2011 (gmt 0)

I am happy to know that webmaster is helping in crawling the pages so faster.

deadsea

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4257070 posted 1:53 pm on Jan 24, 2011 (gmt 0)

Have you started an SEM campaign against this new section of the site? I have seen googlbot go crazy for SEM against large sites. It seems that it is set to do one crawl per keyword that you bid on every day. I've seen it crawl the same landing page that has 100 keywords bid against it 100 times a day. When you are bidding on millions of keywords, the crawl rates can be debilitating.

xnavigator

5+ Year Member



 
Msg#: 4257070 posted 2:03 pm on Jan 24, 2011 (gmt 0)

no i just linked them from other pages

Robert Charlton

WebmasterWorld Administrator robert_charlton us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4257070 posted 1:21 am on Jan 25, 2011 (gmt 0)

no i just linked them from other pages

That explains a lot. They would of course follow the new links.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved