homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Reasons for Low Crawl Rate?

 7:32 am on Nov 12, 2009 (gmt 0)

We have two sites that use the same platform- one of them is getting crawled very regularly, and the other one has barely gotten crawled since October. I have no idea what would cause one to get crawled more, while the other would barely be crawled at all.

Any thoughts? Anything we can do to increase the crawl rate?




 10:42 am on Nov 12, 2009 (gmt 0)

Incoming links from pages with high PR - is the best way to increase speed of crawling.


 10:45 am on Nov 12, 2009 (gmt 0)

Right...but we have the same incoming links that we had 2 months ago, and suddenly our crawl rate tanked- will new links help that, or is it a sign of some other problem?


 11:05 am on Nov 12, 2009 (gmt 0)

From my experience one link with PR 7 or higher enough for dramatically increase speed of crawling for a short period.

How many pages indexed by google on both your sites?


 4:06 pm on Nov 12, 2009 (gmt 0)

One thing that can cause your crawl rate to slow is a poor server response time. If Google starts to think your server has load issues, it will slow down. You can check the Webmaster Tools console Diagnostic->Crawl Stats display to see if there has been a slowdown in your response times.

I speak from experience here. A couple of years ago I changed a site from static pages to dynamically-generated pages so I could automate it. Between the host's poor mySQL performance and my own sloppy programming, it was a dog of a performer and Google slowed down to crawling a handful of pages per day. Retooling my software brought the response times back down to something reasonable (<500ms) and Google restored its crawl rate to "normal" in a couple of weeks.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved