Welcome to WebmasterWorld Guest from 54.146.246.4

Message Too Old, No Replies

Google bot going mad on my site, 10 pages/sec

     

xnavigator

9:49 pm on Jan 23, 2011 (gmt 0)

5+ Year Member



lol since i opened a new section on my website google bot is requesting like 10pages every second all day long.

Funny thing is he is crawling teh same page even if they handle if-modified-since

lucky i have disabled the apache logger... 10gb of logs aren't fun to manage

add: just want to share with you this behavoiour.
add2: bot' ip seem to be only 2 (starting with 66.249)

tedster

11:28 pm on Jan 23, 2011 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



That's not normal googlebot behavior. Have you tested it with this approach: How to Verify Googlebot [webmasterworld.com]?

xnavigator

11:56 pm on Jan 23, 2011 (gmt 0)

5+ Year Member



dns and ip for both match. (it's googlebot)


edit: why do u think it's not normal? (for large website this can only speed up the indexing process... the only problem is it's requesting pages that has already indexed like yesterday)

[edited by: xnavigator at 12:09 am (utc) on Jan 24, 2011]

tedster

12:07 am on Jan 24, 2011 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Ah, I misunderstood. I thought you were getting repeated requests within a second for the same URL - even after serving a 304 response. Well, repeatedly checking on a 304 response doesn't take much time at all - as long as other URLs are also getting requested.

xnavigator

12:10 am on Jan 24, 2011 (gmt 0)

5+ Year Member



yes sorry my bad..

not only 1 url... but every between the ~10000 possibile urls

askbjoernhansen

6:46 am on Jan 24, 2011 (gmt 0)

5+ Year Member



We get about 10 requests a second per domain they crawl. In the webmaster tool that seems to be the highest setting we can set if we set it manually...

xnavigator

11:53 am on Jan 24, 2011 (gmt 0)

5+ Year Member



in my webmaster tools i can set max only 2.2 requests per second.. (while currenty it's crawling 10 per sec)

But anyway I never believed webmaster tool really influenced google :D

ChristeneMint

12:12 pm on Jan 24, 2011 (gmt 0)



I am happy to know that webmaster is helping in crawling the pages so faster.

deadsea

1:53 pm on Jan 24, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Have you started an SEM campaign against this new section of the site? I have seen googlbot go crazy for SEM against large sites. It seems that it is set to do one crawl per keyword that you bid on every day. I've seen it crawl the same landing page that has 100 keywords bid against it 100 times a day. When you are bidding on millions of keywords, the crawl rates can be debilitating.

xnavigator

2:03 pm on Jan 24, 2011 (gmt 0)

5+ Year Member



no i just linked them from other pages

Robert Charlton

1:21 am on Jan 25, 2011 (gmt 0)

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



no i just linked them from other pages

That explains a lot. They would of course follow the new links.