file not found

[stanleytk]

We had performed a major upgrade to our website a couple of months ago. This upgrade had caused most all of the previous pages to become extinct as they were replaced with new pages. When I resubmitted our site to the search engines, many engines said they would reindex the pages and drop the old ones. But as I watch the spiders come in, many of the spiders are still requesting the older pages (i.e. Ultraseek 209.135.65.57) and they are getting the 404 file not found error. We have created our own 404 page to redirect the user to the home page to overcome this problem.
I thought I saw a post indicating how to tell the spiders that the page they are requesting should be removed from their db as it no linger exists. Can anyone help me find that post or let me know if there is a way in which I can do this?
thanks

[Froggyman]

In .htaccess file add the following line:

Redirect 301 /oldpage.html [yourwebsite.com...]

[mivox]

The 301 redirect will also direct your human visitors to the new pages, solving two problems at once.

[Xoc]

If you are giving the engine a true 404 page, with the proper status in the http header, that is the closest thing you can do to tell a spider that the page is gone.

To see the http header:

Start a DOS command prompt
From the command prompt type

telnet [microsoft.com...] 80

(Turn on localecho so you can see what you are doing.

Type Ctrl+] (hold down the Ctrl key and press the right bracket)

Type

set localecho

Press Enter on a blank line

You only have to turn on localecho once.)

type

GET / HTTP/1.1
Host: www.microsoft.com

Press Enter twice