Vimes - 10:12 am on Sep 7, 2008 (gmt 0)

Well I seem to experience a deep crawl on the site just before my GWT updates itself for links extra so I wasn’t surprised to see it as that normally happens for me at the end of each month, the only difference this time was the amount of old URL’s that were requested, that’s why I left it a week before mentioning anything, normally this activity dies off after a day or three and things return to normal and life goes on. But this activity has only just dropped off, over the last 10 days I’ve had just over 25,000 404’s requested from Googlebot. 95% of these are URL’s that disappeared off my site after we restructured the entire website just over 4 years ago, yes we had redirects on these for a while, these were probably removed about 2 years ago.

I recently added a 301 redirect to my root stopping any issues with the www.domain.com./ , I wouldn’t have thought that this would have caused any huge requests for URL’s redirecting to 404 pages as a page that isn’t there just isn’t there.
But I guess it might have channeled the Bot to recheck, I’m checking my 301 redirect logs now and so far haven’t come across any www.doamin.com./ 301 redirects landing to a 404 page.

I really get nervous when googlebot does funky stuff like this, for me its never been a good.

Vimes.