lol since i opened a new section on my website google bot is requesting like 10pages every second all day long.
Funny thing is he is crawling teh same page even if they handle if-modified-since
lucky i have disabled the apache logger... 10gb of logs aren't fun to manage
add: just want to share with you this behavoiour. add2: bot' ip seem to be only 2 (starting with 66.249)
tedster
11:28 pm on Jan 23, 2011 (gmt 0)
That's not normal googlebot behavior. Have you tested it with this approach: How to Verify Googlebot [webmasterworld.com]?
xnavigator
11:56 pm on Jan 23, 2011 (gmt 0)
dns and ip for both match. (it's googlebot)
edit: why do u think it's not normal? (for large website this can only speed up the indexing process... the only problem is it's requesting pages that has already indexed like yesterday)
[edited by: xnavigator at 12:09 am (utc) on Jan 24, 2011]
tedster
12:07 am on Jan 24, 2011 (gmt 0)
Ah, I misunderstood. I thought you were getting repeated requests within a second for the same URL - even after serving a 304 response. Well, repeatedly checking on a 304 response doesn't take much time at all - as long as other URLs are also getting requested.
xnavigator
12:10 am on Jan 24, 2011 (gmt 0)
yes sorry my bad..
not only 1 url... but every between the ~10000 possibile urls
askbjoernhansen
6:46 am on Jan 24, 2011 (gmt 0)
We get about 10 requests a second per domain they crawl. In the webmaster tool that seems to be the highest setting we can set if we set it manually...
xnavigator
11:53 am on Jan 24, 2011 (gmt 0)
in my webmaster tools i can set max only 2.2 requests per second.. (while currenty it's crawling 10 per sec)
But anyway I never believed webmaster tool really influenced google :D
ChristeneMint
12:12 pm on Jan 24, 2011 (gmt 0)
I am happy to know that webmaster is helping in crawling the pages so faster.
deadsea
1:53 pm on Jan 24, 2011 (gmt 0)
Have you started an SEM campaign against this new section of the site? I have seen googlbot go crazy for SEM against large sites. It seems that it is set to do one crawl per keyword that you bid on every day. I've seen it crawl the same landing page that has 100 keywords bid against it 100 times a day. When you are bidding on millions of keywords, the crawl rates can be debilitating.
xnavigator
2:03 pm on Jan 24, 2011 (gmt 0)
no i just linked them from other pages
Robert Charlton
1:21 am on Jan 25, 2011 (gmt 0)
no i just linked them from other pages
That explains a lot. They would of course follow the new links.
