Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Googlebot crawling rate and Panda

         

grimmer

8:15 pm on May 22, 2011 (gmt 0)

10+ Year Member



We noticed that googlebot is crawling much slower on our site then it did couple months ago. This month, the average number of pages crawled is only half of the size two months ago. We were affected by Panda quite bad, and we are not sure whether because of this, google is crawling much slower on the site, does anyone notice the crawling rate difference since Panda?
Also, the WMT provides options to adjust the crawl speed, does it actually work? We have increased the crawling speed 100%, but do not see any difference.

deadsea

11:43 am on May 23, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



As far as I can tell, Googlebot crawl rate is a straightforward combination of site performance and pagerank.

We see the crawl rate go up whenever our site performance improves. In fact page download speed and number of pages download by Googlebot are inversely proportional.

Sites with higher pagerank also get crawled more. Even within a site, the pages that are higher pagerank get crawled more often. In fact, I use crawl rate of a page instead of pagerank when I do SEO experiments. You can measure it in a couple weeks and it is very granular, whereas it takes months for a toolbar update and you only get one of 12 values.

One theory with panda is that it reduced spammy sites such that they don't pass pagerank through their links. Sites that get links from those sites seem to be hit by panda. Not because there is anything wrong with the site per say, but because they lost of lot of link juice. That theory would fit very well with the reduced crawl rate that you are seeing.

Adjusting the crawl rate upwards doesn't seem to have much effect in my experience. I've always been able to dial the speed down, but "100%" and "let google decide" appear to be the same thing.

Even if you could set it higher, the tail does not wag the dog. You see a higher crawl rate when you get more traffic because you have higher pagerank. Setting the crawl rate faster does not cause your pagerank to increase or your site traffic to increase.

walkman

12:21 pm on May 23, 2011 (gmt 0)



grimmer, like deadsea said. You can increase it but it will not exceed what Google has it as max (based on your PR for the most part). Matt Cutts said that after Panda hit sites will be crawled slower but honestly I don't believe him at all, he's just a Google public relation boy now.

My site was hit heavily on 2/24 and has gone down traffic wise even more since then but my crawling is at least the same, even though I have 50% less pages. My pages are not thin though and are linked very well through the site.