Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

When G sets crawl rate - what is a good number?

         

latimer

9:44 pm on Jan 14, 2010 (gmt 0)

10+ Year Member



G wmt says we have approx 70% of total pages in index. We "let G determine my crawl rate", but wonder about the 20 secs between requests on a site with 150,000 pages. Does tthe crawl rate number give any hints as to our sites quality or trust in G's algo, or our abiilty to get higher percentage of pages indexed?

rainborick

1:06 am on Jan 15, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The crawl rate is the speed or frequency with which Google crawls your site. The setting is largely there to give webmasters a chance to slow their crawl rate if crawling starts to be a load problem, or to speed it back up if Google has reduced it due to slow response times or other problems.

The number of pages indexed is the crawl depth. This is largely determined by PageRank and the internal link structure of the site and is not affected by the Webmaster Tools setting.

AnkitMaheshwari

5:34 am on Jan 15, 2010 (gmt 0)

10+ Year Member



For one of our site, we increased the crawl rate to test and see if it does help in improving index, however resulted in a lot of loading issue reports in WMT for network timeouts, unreachable or simply 404 when the URL's were working fine.

Once we set it back to "Let G Decide the speed" things were normal within about 15 days.

Note: No increase in indexing achieved. IMO G only was able to understand that our servers were slow or down :(