We just re-launched out portal with completely new file names. Our custom 404 page is served when we have any requests for old files. Now I notice that for example MSN is caching our 404 page! If this continues we will end up having thousands of pages in the indexes of search engines and half of the pages will be identical 404 pages...
Please help me prevent that, we must have done something wrong setting it up. Users are being redirected to the 404 page, is that the correct way to do it?
How does a typical crawer react to a 404 page? And how do they react to a 500 page? Would these 2 responses normally cause the file to be removed from the index?
Very grateful for any help you can give. Have a nice day!
Did you do the server header check on the 404 page? Or, on a non-existent page? The non-existent page is probably returning a 404 and the custom 404 page is probably returning a 200. This is usually the case 8 out of 10 times.
Msg#: 4341 posted 10:13 am on Nov 27, 2005 (gmt 0)
I am back... It seems the search engines igonore the META tags mentioned in my previous post. Instead of seeing a decreased amount of pages in Google, an allinurl query brings back several hundred new pages each day.
If the costum 404 page brings back a 200 in the header check tool, how can we make sure it gives a 404? Users trying to view one of our old pages are re-directed to the custom 404 page. Are there any other ways to server that page than using re-directs?
I cannot understand what we are doing wrong and I am not knowledgeable in the technical details of portal. Any of you have any ideas? Thanks for any advice.