Welcome to WebmasterWorld Guest from 54.166.148.252

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Low Crawl Rate warning in Webmaster Tools

     
4:46 am on Aug 20, 2009 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 21, 2003
posts: 73
votes: 0


Hi,

I used to always set my crawl rate by let Google determine by default which they said "Recommended". Then several months ago I noticed that crawl rates from Google was drastically dropped, so I change the option by set my own custom crawl rates by increased them from the slide bar to 250% more. It was working fine since then.

However, Google send me a warning message that they were unable to crawl my site due to 'Low Crawl Rate' setting few days ago. And this is also impact to the number of pages indexed showing in WMT as well.

So I checked my custom crawl rate slider bar again, but it was almost to the maximum speed that I can adjust which I can not go much further. I just can't figure it out how this speed can be considered as a "Low crawl rate" from Google according to their warning message.

Anyone experience the same situation here? Appreciate any comment.

thanks in advance.

6:18 pm on Aug 20, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Sounds like some kind of bug, doesn't it.

And this is also impact to the number of pages indexed showing in WMT as well.

I'm not sure why you feel that the crawl speed message is CAUSING a drop in the number of pages indexed. They're not the same thing.

If I were in your situation, I would probably experiment with returning to letting Google decide the crawl speed.

7:08 pm on Aug 20, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 15, 2003
posts:926
votes: 21


When I had a crawl rate issue about a year ago, it was due to sluggish response times on my server. I improved the underlying software and that solved the problem for me. You might check the Crawl Stats section in WMT and look for spikes in the downloading times graph. Google will override your WMT setting and throttle back your crawl rate if it sees a sustained period of slow responses.