| 5:07 pm on Nov 18, 2005 (gmt 0)|
Is the server really returning a 404 not found error, or is it actually returning a 302 found or other header? If the server is returning a 302 or 200 then the spider may well cache the page.
Try the server header check tool [webmasterworld.com] for a non-existent page on your site to see what is happening.
| 5:51 pm on Nov 18, 2005 (gmt 0)|
Many thanks for such a quick reply!
This is the response from the header check:
HTTP/1.1 404 Not Found
Date: Fri, 18 Nov 2005 17:45:23 GMT
So everything looks ok to me. Now, what is the next step in trying to solve this problem?
Thanks again, I am very grateful.
| 7:13 pm on Nov 20, 2005 (gmt 0)|
I just hope that anyone else here has an ideas about why search engines cache our custom made 404 page. Ciao!
| 7:20 pm on Nov 20, 2005 (gmt 0)|
Did you do the server header check on the 404 page? Or, on a non-existent page? The non-existent page is probably returning a 404 and the custom 404 page is probably returning a 200. This is usually the case 8 out of 10 times.
| 7:48 pm on Nov 20, 2005 (gmt 0)|
pageoneresults, you are absolutely right! Thanks a lot. So all we need to do is to block robots from indexing the page, just like one does with normal pages?
<META NAME="msnbot" CONTENT="noarchive"> for MSN
<META NAME="ROBOTS" CONTENT="NOARCHIVE"> For all other robots
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> Another way to prevent indexing
Plus a robots.txt file of course.
| 10:13 am on Nov 27, 2005 (gmt 0)|
I am back... It seems the search engines igonore the META tags mentioned in my previous post. Instead of seeing a decreased amount of pages in Google, an allinurl query brings back several hundred new pages each day.
If the costum 404 page brings back a 200 in the header check tool, how can we make sure it gives a 404? Users trying to view one of our old pages are re-directed to the custom 404 page. Are there any other ways to server that page than using re-directs?
I cannot understand what we are doing wrong and I am not knowledgeable in the technical details of portal. Any of you have any ideas? Thanks for any advice.