homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

file not found

 1:09 pm on Apr 17, 2001 (gmt 0)

We had performed a major upgrade to our website a couple of months ago. This upgrade had caused most all of the previous pages to become extinct as they were replaced with new pages. When I resubmitted our site to the search engines, many engines said they would reindex the pages and drop the old ones. But as I watch the spiders come in, many of the spiders are still requesting the older pages (i.e. Ultraseek and they are getting the 404 file not found error. We have created our own 404 page to redirect the user to the home page to overcome this problem.
I thought I saw a post indicating how to tell the spiders that the page they are requesting should be removed from their db as it no linger exists. Can anyone help me find that post or let me know if there is a way in which I can do this?



 12:42 am on Apr 18, 2001 (gmt 0)

In .htaccess file add the following line:

Redirect 301 /oldpage.html [yourwebsite.com...]


 12:50 am on Apr 18, 2001 (gmt 0)

The 301 redirect will also direct your human visitors to the new pages, solving two problems at once.


 2:58 am on Apr 18, 2001 (gmt 0)

If you are giving the engine a true 404 page, with the proper status in the http header, that is the closest thing you can do to tell a spider that the page is gone.

To see the http header:

Start a DOS command prompt
From the command prompt type

telnet [microsoft.com...] 80

(Turn on localecho so you can see what you are doing.

Type Ctrl+] (hold down the Ctrl key and press the right bracket)


set localecho

Press Enter on a blank line

You only have to turn on localecho once.)


GET / HTTP/1.1
Host: www.microsoft.com

Press Enter twice

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved