A drop in googlebot activity means?

[kidder]

Just over a week ago I've introduced some changes to my forum, it's very well established (6 years)with plenty of activity and solid rankings. So we setup a blog that acts as a sort of retail shopping area for the members & then blocked it via robots.txt - We then added several text links in the header to the various sections of the shop. So it was business as usual up until about 2 days ago when I've noticed a decline in crawl activity on the site. Rankings and traffic are holding and maybe I'm paranoid but where there is smoke there just might be fire? Googlebot is still crawling the site it's just not eating up the pages like it was a week ago. Any thoughts.

[tedster]

I'm not sure it means anything. Spidering goes through cycles of intensity and then drops to a low-level at other times. Mostly I think it has to do with projects and code revisions going on in the back room at Google. The time I'd worry is when everyone on the forums is talking about their server getting hammered by googlebot and I'm still seeing less than usual levels of spidering.

However, if you can correlate the robots.txt changes with the drop in spidering, it would be good to doublecheck the cyntax to be sure it's doing what you intend. There's a very useful tool inside Webmaster Tools accounts that helps with that,

[edited by: tedster at 7:41 am (utc) on Mar. 7, 2008]

[kidder]

Thanks once again Ted - I ran a couple of tests today and found I could get new pages ranking at the top of the serps in less than 10 mins in some cases. Once came in at 7 min and the other at 11 min - I guess that tells me I should spend more time writing rather than worrying. I did run the robots.txt check and it reported the directory as blocked. As a double measure I no follow'd the links just to be on the safe side. I've lost sites from the index before and once you've been hit like that your always looking over your shoulder.

[Asia_Expat]

I noticed the same thing a few days ago on my main site. Usually there are around 200 'guests' browsing the forum in a one hour period but this has dropped dramatically to around 60. I am not seeing a drop in real traffic though, as reported by analytics.

I'm not worried about this and suspect it's bot throttling because my server went down for around 10 hours a few days ago because my host updated the platform and broke my PHP scripts.

[ecmedia]

Noticed slightly slower crawling activity from G during last 10 days or so but it seems it is returning to normal now.

[kidder]

Yes today we are back to "above normal" levels of bot activity..