jim_w - 3:40 pm on Dec 9, 2003 (gmt 0)

In my case with the 404’s I believe it is because some bot, that shall remain nameless, was having a problem with 301’s. Instead of just going to the index.html, or whatever, and following the links, they keep some kind of cache of where they think the pages are and what the URL is. When that cache is really outdated plus the problem with the 301’s, it has created a mess for me. Do doubt it was just my luck to make these changes at the wrong time, but how was I to know it was the wrong time. Which makes my point about NOT going live until you, blah, blah, blah.

This could be one reason why there are now 404's showing up in the searches. And since the last update of the nameless bot, it is now returning these pages instead of the pages that should be returned based on what the webmaster expected the nameless search engine to return. Before the update, I never saw a 404.

My server does return the proper header, I have checked. This is clearly a problem ONLY with the nameless bot due to the fact that all the other bots have done it correctly. There is only one place to place the blame. Well, unless it is my fault for changing the names of some of my pages for better file management reasons. Using the theory that the more complex something is, the higher the probability for an error, and case studies performed by major universities indicate that the error rate it is exponential. In the case of a search engine, you have 2 pieces of code running. The part that fetches URL’s and the part that searches the human query. So the probability for error is more than twice as likely.

claus, are you saying that the nameless search engine is not susceptible to errors in the coding?