Welcome to WebmasterWorld Guest from 23.20.221.93

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Googlebot is crashing my site

googlebot is on an infinite loop of one of my pages, help!

     
4:46 pm on Jan 14, 2007 (gmt 0)

New User

10+ Year Member

joined:May 5, 2005
posts:40
votes: 0


I run a site that has over 2000 pages. Normally we are crawled thoroughly and wonderfully everyday. We have five sections of archives pages. Last night googlebot got stuck on the first page of each of our five entry archive pages. Googlebot is stuck on an infinite loop of crawling those pages and we can't stop it. It is not going anywhere else (other than the homepage, which it goes to every 5 minutes) on the site, just those 5 pages and the homepage. We put a meta name='robots' content='noindex,nofollow' tag, it didn't stop it, we put a meta name='googlebot' content='noindex, nofollow', it is still stuck and we don't know what to do. It is not following any of the links on any of our pages and the constant loop is causing us to lose memory and we keep getting shut down. Any idea what the heck is going on?
9:35 pm on Jan 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


Whats the ip address its coming from?
9:49 pm on Jan 14, 2007 (gmt 0)

New User

10+ Year Member

joined:May 5, 2005
posts:40
votes: 0


My programmer isn't available at the moment, but when we checked the IP earlier we found that it was a real googlebot IP, not a fake or hacker.

We are not sure how long this has been going on, just caught it last night because our site went down, but since we restarted the server our programmer has been watching the logs and noticed what I explained in the initial post.

10:10 pm on Jan 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


Go to this link:

[google.com...]

and contact google. Follow the instructions. They might not respond until tommorrow though.

10:17 pm on Jan 14, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


You also might want to deny the ip for now as well until its resolved.