joined:July 20, 2001
Over the last few months Googlebot has found it necessary to spider an increasing number of pages on my sites more than once in succession.
When it first began doing this it would just happen for a few hours in the odd day, taking pages twice in succession. It has now reached the point where this takes place all the time, and yesterday I was dismayed to see it visit the same page 4 times in succession, and today 5!
It uses the same if-modified-since date for each request to a page. I only change this date when the page content is updated. Some pages even have 301 redirects and Googlebot will still visit these 3 times in succession.
I did raise this in Google's help forums and also found another guy experiencing the same problem, but found no answers.
Has anyone else experienced this and do you have any idea what purpose it may serve?