Msg#: 4023614 posted 7:32 am on Nov 12, 2009 (gmt 0)
We have two sites that use the same platform- one of them is getting crawled very regularly, and the other one has barely gotten crawled since October. I have no idea what would cause one to get crawled more, while the other would barely be crawled at all.
Any thoughts? Anything we can do to increase the crawl rate?
Msg#: 4023614 posted 4:06 pm on Nov 12, 2009 (gmt 0)
One thing that can cause your crawl rate to slow is a poor server response time. If Google starts to think your server has load issues, it will slow down. You can check the Webmaster Tools console Diagnostic->Crawl Stats display to see if there has been a slowdown in your response times.
I speak from experience here. A couple of years ago I changed a site from static pages to dynamically-generated pages so I could automate it. Between the host's poor mySQL performance and my own sloppy programming, it was a dog of a performer and Google slowed down to crawling a handful of pages per day. Retooling my software brought the response times back down to something reasonable (<500ms) and Google restored its crawl rate to "normal" in a couple of weeks.