Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How to make googlebot visit your pages at an increased rate

         

Sgt_Kickaxe

4:27 am on Jul 18, 2011 (gmt 0)



Make a change to some text which is visible sitewide, a welcome message perhaps or other text which can be found on all pages. It also works if you add sitewide text, even a sentence or two. Googlebot will notice the change to your otherwise static content and tear through pages at an increased rate, often 300% to 500% above the norm, for a short period of time (5 minutes?). If you're used to 600 pageviews an hour, including googlebot, you'll see 3-5x that in 5 minutes, all google.

Make sure your stats can track googlebot, I first noticed it because googlebot was creating a lot of cached pages quickly as I tweaked some welcome text. It was repeatable.

If I had to guess it seems that Googlebot has template characteristics and sitewide text zero'd out but when a change is made to these areas Google wants to know immediately how widespread it is.

This is just a little tidbit of information with no guarantees. I would not recommend abusing it but it's worth tucking away to make Googlebot jump when you want it to.

Note: It doesn't seem to work unless you have the 'let Google decide' setting in your GWT for crawl rate.

tedster

5:16 pm on Jul 18, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Interesting observation. For some reason it reminds me of Matt Cutts recent comment that Google now has the capacity to crawl every [page in their index within a day - but they don't because it would be a resource drain on others.