tedster - 8:58 pm on Jan 15, 2006 (gmt 0)

Here's one I've seen hurt lots of sites -- make sure your "404" responses REALLY return a 404 HTTP header. Many sites try to redirect their 404 traffic to a custom page or their Home Page. But when they set it up, they end up returning a 302 header or even *shudder* a 200.

After a while, the Google index can get filled with bad urls that are all duplicate copies of the custom page or Home Page. This can really wreak havoc. I have even heard of a darker practice that goes looking for this issue and intentionally helps Google to "find" lots of those bad urls.

Even without someone intentionally feeding googlebot those bad urls, the spider seems to be quite creative in requesting some funky addresses all on its own. If you don't feed it a 404 server header, you're off and running.

Any doubts about your custom 404 set up? You can check the server header [searchengineworld.com] for your should-be-404 url on our sister site, SearchEngineWorld.