Google crawling unnecessarily

[cyberdyne]

I recently allowed Google to start crawling my guest book whereas previously it was blocked via robots.txt. I've also recently removed the .rss facility from my guest book (unrelated reason).

The problem is that Google is now incessantly crawling non-existent .rss pages in the format: /guesbook/rss.php?entry={number}
It is even crawling numbers as high as 99999 which never even existed in the first place.

How can I stop this from happening please? Even though Google is being served 404's, the huge number of them is using considerable bandwidth.

Thanks in advance

[tedster]

You either block the URL pattern or you let googlebot do it's thing. The pattern you describe is not likely to go on very long, however. How much bandwidth can a bunch of 404 responses use up, after all?

[cyberdyne]

Fair enough, thanks

[g1smd]

Does the 404 page for a non-existent entry have a "next" link pointing to a higher number?

If it has, Google could go on forever.

The same problem affects online calenders. I once saw a site with a calendar of events spanning less than a decade but where Google had crawled several thousand years into the future and into the past.

[cyberdyne]

No links at all, no. It auto-refreshes to the index after a few seconds but has no links.

Thanks for the thought though.

[lucy24]

I once saw a site with a calendar of events spanning less than a decade but where Google had crawled several thousand years into the future and into the past.

Please! Not while I'm drinking my tea!

There's a scattering of unrelated posts that mention Google testing whether your site ever returns a 404. Apparently one way is to feed in an obviously bogus query.

Would be nice, wouldn't it, if the Crawl Errors distinguished between your own bona fide intentional links... and the ones they made up to test you.

[manny123]

Any chance it's not actually a google bot? Run a reverse DNS or a whois at ARIN on the IP address just to make sure. The sequential numbering seems a bit suspect given that the links never existed.

[Pjman]

I had the same thing happen to me. I had a script that listed posts by numeric ID. Googlebot started throwing in random numbers and indexing them, even though there were no links leading them their. G indexed 40,000 non-existent URLs. I think it was because it was getting a 200 header code when you punched any number in there. I feel it was a big part of why I was Pandalized.

I added a few lines to the code to return 404s if you put in a bogus number. The index bogus pages dropped from 40,000 to actual 3,000 posts that exist.

I'm still panadalized, but I only went through one Panda refresh since. I hear that you need 2 Panda runs to get out of the Panda box. Finger crossed.

[g1smd]

About this time last year I fixed a site with a similar problem and dropped their indexing from ~25 000 bogus URLs to ~75 real pages. It took exactly six months for the WMT reports to catch up with reality.

[cyberdyne]

It's definitely Google, yes, and the activity appears to have eased off since I added a line to my robots forbidding crawling of the offending url's. Doesn't mean it'll stop of course but for now at least it's slowed considerably.
Was having a similar issue with profile pages on a forum too but again that has eased by adding a specific line to robots.txt.