| 11:28 pm on Jan 23, 2011 (gmt 0)|
That's not normal googlebot behavior. Have you tested it with this approach: How to Verify Googlebot [webmasterworld.com]?
| 11:56 pm on Jan 23, 2011 (gmt 0)|
dns and ip for both match. (it's googlebot)
edit: why do u think it's not normal? (for large website this can only speed up the indexing process... the only problem is it's requesting pages that has already indexed like yesterday)
[edited by: xnavigator at 12:09 am (utc) on Jan 24, 2011]
| 12:07 am on Jan 24, 2011 (gmt 0)|
Ah, I misunderstood. I thought you were getting repeated requests within a second for the same URL - even after serving a 304 response. Well, repeatedly checking on a 304 response doesn't take much time at all - as long as other URLs are also getting requested.
| 12:10 am on Jan 24, 2011 (gmt 0)|
yes sorry my bad..
not only 1 url... but every between the ~10000 possibile urls
| 6:46 am on Jan 24, 2011 (gmt 0)|
We get about 10 requests a second per domain they crawl. In the webmaster tool that seems to be the highest setting we can set if we set it manually...
| 11:53 am on Jan 24, 2011 (gmt 0)|
in my webmaster tools i can set max only 2.2 requests per second.. (while currenty it's crawling 10 per sec)
But anyway I never believed webmaster tool really influenced google :D
| 12:12 pm on Jan 24, 2011 (gmt 0)|
I am happy to know that webmaster is helping in crawling the pages so faster.
| 1:53 pm on Jan 24, 2011 (gmt 0)|
Have you started an SEM campaign against this new section of the site? I have seen googlbot go crazy for SEM against large sites. It seems that it is set to do one crawl per keyword that you bid on every day. I've seen it crawl the same landing page that has 100 keywords bid against it 100 times a day. When you are bidding on millions of keywords, the crawl rates can be debilitating.
| 2:03 pm on Jan 24, 2011 (gmt 0)|
no i just linked them from other pages
| 1:21 am on Jan 25, 2011 (gmt 0)|
|no i just linked them from other pages |
That explains a lot. They would of course follow the new links.