Msg#: 4257070 posted 12:07 am on Jan 24, 2011 (gmt 0)
Ah, I misunderstood. I thought you were getting repeated requests within a second for the same URL - even after serving a 304 response. Well, repeatedly checking on a 304 response doesn't take much time at all - as long as other URLs are also getting requested.
Msg#: 4257070 posted 1:53 pm on Jan 24, 2011 (gmt 0)
Have you started an SEM campaign against this new section of the site? I have seen googlbot go crazy for SEM against large sites. It seems that it is set to do one crawl per keyword that you bid on every day. I've seen it crawl the same landing page that has 100 keywords bid against it 100 times a day. When you are bidding on millions of keywords, the crawl rates can be debilitating.