Welcome to WebmasterWorld Guest from 54.226.32.234

Message Too Old, No Replies

Low Crawl Rate warning in Webmaster Tools

     

Cyclob

4:46 am on Aug 20, 2009 (gmt 0)

10+ Year Member



Hi,

I used to always set my crawl rate by let Google determine by default which they said "Recommended". Then several months ago I noticed that crawl rates from Google was drastically dropped, so I change the option by set my own custom crawl rates by increased them from the slide bar to 250% more. It was working fine since then.

However, Google send me a warning message that they were unable to crawl my site due to 'Low Crawl Rate' setting few days ago. And this is also impact to the number of pages indexed showing in WMT as well.

So I checked my custom crawl rate slider bar again, but it was almost to the maximum speed that I can adjust which I can not go much further. I just can't figure it out how this speed can be considered as a "Low crawl rate" from Google according to their warning message.

Anyone experience the same situation here? Appreciate any comment.

thanks in advance.

tedster

6:18 pm on Aug 20, 2009 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Sounds like some kind of bug, doesn't it.

And this is also impact to the number of pages indexed showing in WMT as well.

I'm not sure why you feel that the crawl speed message is CAUSING a drop in the number of pages indexed. They're not the same thing.

If I were in your situation, I would probably experiment with returning to letting Google decide the crawl speed.

rainborick

7:08 pm on Aug 20, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



When I had a crawl rate issue about a year ago, it was due to sluggish response times on my server. I improved the underlying software and that solved the problem for me. You might check the Crawl Stats section in WMT and look for spikes in the downloading times graph. Google will override your WMT setting and throttle back your crawl rate if it sees a sustained period of slow responses.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month