Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: phranque
Please help me prevent that, we must have done something wrong setting it up. Users are being redirected to the 404 page, is that the correct way to do it?
How does a typical crawer react to a 404 page? And how do they react to a 500 page? Would these 2 responses normally cause the file to be removed from the index?
Very grateful for any help you can give. Have a nice day!
Try the server header check tool [webmasterworld.com] for a non-existent page on your site to see what is happening.
HTTP/1.1 404 Not Found
Date: Fri, 18 Nov 2005 17:45:23 GMT
So everything looks ok to me. Now, what is the next step in trying to solve this problem?
Thanks again, I am very grateful.
<META NAME="msnbot" CONTENT="noarchive"> for MSN
<META NAME="ROBOTS" CONTENT="NOARCHIVE"> For all other robots
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> Another way to prevent indexing
Plus a robots.txt file of course.
If the costum 404 page brings back a 200 in the header check tool, how can we make sure it gives a 404? Users trying to view one of our old pages are re-directed to the custom 404 page. Are there any other ways to server that page than using re-directs?
I cannot understand what we are doing wrong and I am not knowledgeable in the technical details of portal. Any of you have any ideas? Thanks for any advice.