Welcome to WebmasterWorld Guest from 107.20.54.98

Message Too Old, No Replies

Reasons for Low Crawl Rate?

     
7:32 am on Nov 12, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:June 2, 2008
posts: 49
votes: 0


We have two sites that use the same platform- one of them is getting crawled very regularly, and the other one has barely gotten crawled since October. I have no idea what would cause one to get crawled more, while the other would barely be crawled at all.

Any thoughts? Anything we can do to increase the crawl rate?

Thanks!

10:42 am on Nov 12, 2009 (gmt 0)

New User

5+ Year Member

joined:May 6, 2009
posts:21
votes: 0


Incoming links from pages with high PR - is the best way to increase speed of crawling.
10:45 am on Nov 12, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:June 2, 2008
posts:49
votes: 0


Right...but we have the same incoming links that we had 2 months ago, and suddenly our crawl rate tanked- will new links help that, or is it a sign of some other problem?
11:05 am on Nov 12, 2009 (gmt 0)

New User

5+ Year Member

joined:May 6, 2009
posts:21
votes: 0


From my experience one link with PR 7 or higher enough for dramatically increase speed of crawling for a short period.

How many pages indexed by google on both your sites?

4:06 pm on Nov 12, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 15, 2003
posts:904
votes: 5


One thing that can cause your crawl rate to slow is a poor server response time. If Google starts to think your server has load issues, it will slow down. You can check the Webmaster Tools console Diagnostic->Crawl Stats display to see if there has been a slowdown in your response times.

I speak from experience here. A couple of years ago I changed a site from static pages to dynamically-generated pages so I could automate it. Between the host's poor mySQL performance and my own sloppy programming, it was a dog of a performer and Google slowed down to crawling a handful of pages per day. Retooling my software brought the response times back down to something reasonable (<500ms) and Google restored its crawl rate to "normal" in a couple of weeks.