Msg#: 3219061 posted 4:46 pm on Jan 14, 2007 (gmt 0)
I run a site that has over 2000 pages. Normally we are crawled thoroughly and wonderfully everyday. We have five sections of archives pages. Last night googlebot got stuck on the first page of each of our five entry archive pages. Googlebot is stuck on an infinite loop of crawling those pages and we can't stop it. It is not going anywhere else (other than the homepage, which it goes to every 5 minutes) on the site, just those 5 pages and the homepage. We put a meta name='robots' content='noindex,nofollow' tag, it didn't stop it, we put a meta name='googlebot' content='noindex, nofollow', it is still stuck and we don't know what to do. It is not following any of the links on any of our pages and the constant loop is causing us to lose memory and we keep getting shut down. Any idea what the heck is going on?
Msg#: 3219061 posted 9:49 pm on Jan 14, 2007 (gmt 0)
My programmer isn't available at the moment, but when we checked the IP earlier we found that it was a real googlebot IP, not a fake or hacker.
We are not sure how long this has been going on, just caught it last night because our site went down, but since we restarted the server our programmer has been watching the logs and noticed what I explained in the initial post.