homepage Welcome to WebmasterWorld Guest from 54.211.138.180
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
file not found
stanleytk




msg:397353
 1:09 pm on Apr 17, 2001 (gmt 0)

We had performed a major upgrade to our website a couple of months ago. This upgrade had caused most all of the previous pages to become extinct as they were replaced with new pages. When I resubmitted our site to the search engines, many engines said they would reindex the pages and drop the old ones. But as I watch the spiders come in, many of the spiders are still requesting the older pages (i.e. Ultraseek 209.135.65.57) and they are getting the 404 file not found error. We have created our own 404 page to redirect the user to the home page to overcome this problem.
I thought I saw a post indicating how to tell the spiders that the page they are requesting should be removed from their db as it no linger exists. Can anyone help me find that post or let me know if there is a way in which I can do this?
thanks

 

Froggyman




msg:397354
 12:42 am on Apr 18, 2001 (gmt 0)

In .htaccess file add the following line:

Redirect 301 /oldpage.html [yourwebsite.com...]

mivox




msg:397355
 12:50 am on Apr 18, 2001 (gmt 0)

The 301 redirect will also direct your human visitors to the new pages, solving two problems at once.

Xoc




msg:397356
 2:58 am on Apr 18, 2001 (gmt 0)

If you are giving the engine a true 404 page, with the proper status in the http header, that is the closest thing you can do to tell a spider that the page is gone.

To see the http header:

Start a DOS command prompt
From the command prompt type

telnet [microsoft.com...] 80

(Turn on localecho so you can see what you are doing.

Type Ctrl+] (hold down the Ctrl key and press the right bracket)

Type

set localecho

Press Enter on a blank line

You only have to turn on localecho once.)

type

GET / HTTP/1.1
Host: www.microsoft.com

Press Enter twice

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved