I just found several 500 errors on a four year old site that was PR0'd (former PR 4/5) and completely dropped from the google index/directory several weeks ago. Googlebot still visits every couple of days, but only the home page and robots.txt page. The 500 errors were mostly due to an ad running on several pages where the link had a "space" after the "href=" ". Thinking back this "space" was probably added in the time range soon before the site was dropped.
Is it likely I've found the reason it was dropped?
Msg#: 29393 posted 5:26 am on May 15, 2005 (gmt 0)
when it gets a 500 it assumes the server is down so it keeps trying, if it goes on for a while then the page gets put on the back shelf for a while (not dropped completely from the index) if the server comes back up no 500 then the page recovers. I imagine if it went on for a long long time then they would just drop it.
I don't 'know' wether this is the case - just the behavior I've seen - walkman for instance.