Should I use 404 or 410 for GONE for HTTP 1.0 search spiders?

Teoma, Ask.com and almaden.ibm.com seem to index anyway

[Wizcrafts]

I think that Teoma, ask.com and almaden.ibm.com indexing spiders are ignoring my 410 page, for a sub-site and pages that are long gone. They all have HTTP 1.0 as the GET protocol. I keep getting visitors referred by ask or teoma, looking for pages that I marked as 410, 3/4 of a year ago. Should I just mark them as 404 for these HTTP 1.0 spiders? Google does recognize my 410s.

Thanks, Wiz

[davidpbrown]

Obvious, but have you checked the header..
[searchengineworld.com ]

[Wizcrafts]

David;
I just ran the header check with this result:
HTTP/1.1 410 Gone

Do the spiders in question recognize 410 as a valid response for GONE?

Wiz

[davidpbrown]

Certainly I'd expect them to. 410 is a valid HTTP 1.0 code. I'm surprised only by your suggesting Google hasn't acknowledged them. Others can be very slow to catchup and I wouldn't think 6 months+ is unusual.
I'm not expert enough to know whether you need to explicitly suggest HTTP 1.0 410 rather than HTTP 1.1 410. Wouldn't have expected so though.

[Wizcrafts]

My 410 is not specific to any protocol. It simply rues that any request for the removed content is redirected to my 410 page.

Google does recognize my 410, and does not index these pages any more. It only seems to be ask.com, teoma, and ibm that keep the gone pages in their listings, hence sending unqualified traffic to me.

For example: this is from this AM:

wfp2.almaden.ibm.com - - [03/May/2004:22:27:34 -0400] "GET /leatherworks/business4sale.html HTTP/1.0" 410 1619 "-" "http://www.almaden.ibm.com/cs/crawler  [c01]"

And this is from Teoma, a couple of days ago:
egspd447.teoma.com - - [01/May/2004:14:34:27 -0400] "GET /robots.txt HTTP/1.0" 200 1829 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"

egspd447.teoma.com - - [01/May/2004:14:43:00 -0400] "GET /leatherworks/index.html HTTP/1.0" 410 1618 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)"

Both spiders looked at my Robots.txt, where they should have read Disallow: /leatherworks/
and
Disallow: /leatherworks/business4sale.html

Yet, despite the directory and file being set to not be indexed, they looked for it anyway, as they do every month, got my 410, and apparently, ignored it.

Wiz

[edited by: Wizcrafts at 3:03 pm (utc) on May 4, 2004]

[davidpbrown]

<edited to catchup..>

ah, sorry misread that. Yes, I think they may be stagnent but don't know for sure.

Your edit then suggests that they are getting the 410 code so there maybe little more you can do.

I can only suggest maybe direct manual removal requests to each of them.

brain over

[edited by: davidpbrown at 3:08 pm (utc) on May 4, 2004]

[jdMorgan]

This is a good question, as it highlights that HTTP/1.0 and HTTP/1.1 are different, and that the 404-Not Found response, being ambiguous, was extended with 410-Gone in HTTP/1.1.

I use 404 for all HTTP/1.0 requests, since 410 was added in HTTP/1.1

Be aware that Ink takes about a year to give up on requesting pages, and Ask takes awhile too, though I'm not sure how long.

The snippet below returns an explicit 410-Gone for removed pages in response to an HTTP/1.1+ request. If the request is HTTP/1.0, the code does nothing; The server will simply return the default 404 response.

# Respond with 410-Gone status to HTTP/1.1 requests for removed resources.
RewriteCond %{SERVER_PROTOCOL} ^HTTP/(1\.[1-9]¦[2-9]\.[0-9])
RewriteCond %{REQUEST_URI} ^/(announcements¦whatsnew¦weather)\.html$ [OR]
RewriteCond %{REQUEST_URI} ^/events/03_(tickets¦info)\.html$ [OR]
RewriteCond %{REQUEST_URI} ^/(contact_us/¦elections/)
RewriteRule .* - [G]

Jim

[Wizcrafts]

Thanks Jim. I suspected something like this was the problem.

Wiz

[davidpbrown]

doh.. sorry to mislead, of course jdMorgan is correct 410 is 1.1 not 1.0.

Think I need to drink more coffee.
:)

[Wizcrafts]

I just finished my second cup, and my eyes are staring to focus better! Coffee is a major Apache tool!

I have borrowed JD's example and now limit the 410 response to HTTP 1.1.0 and newer.

This is a minor matter, but I couldnt understand why these spiders read robots.txt, ignored the directive to disallow the directory, then kept coming back to look for gone files. I tried contacting Teoma/Ask, but they refused to assist me because I am not a paying member.

Wiz

[guillermo5000]

What is the benefit of serving a 410 (when possible) instead of a 404?

On another somewhat related subject: Is Almaden a beneficial bot? I have banned them for unusually bad behavior on my sites.

Thanks for any replies!

[Wizcrafts]

What is the benefit of serving a 410 (when possible) instead of a 404?

404 means the filename was not found at the specified location. That could be because it was renamed, or relocated to another directory, or sub-site, hence, may still exist somewhere else on the server, but we forgot to issue a 301 Moved response for it.

410 means Gone Permanently, don't bother looking for it again, delete it from your index of this website.

I use a 410 response for directories and files that have been deleted, as opposed to those that have been renamed or relocated to other directories. Unfortunately, it appears that some search engine spiders are using a server protocol that does not recognize 410 as a valid response, and they keep looking for long gone directories and files.

IHTH, Wiz

[jdMorgan]

To expand on Wizcraft's answer, a 404-Not Found response means just that, and only that: The requested resource was not found. This could be permanent because the resource was removed intentionally, or it could be temporary due to a Webmaster error, a database problem, or some other cause that may soon be corrected.

As a result of this ambiguous response, some search engines will keep trying requests for resources that return a 404 response, on the assumption that the problem may be temporary and may be corrected so that the resource will become available again. In cases where a 404 is caused by a temporary error, they are doing us a big favor by retrying, but in other cases, it's just a waste of bandwidth.

The 410-Gone response was added to HTTP/1.1 in order to remove this ambiguity.

Jim

[guillermo5000]

I see. I will use that! Thank you!

Any opinion on the Almaden bot mentioned above?

Thanks again.

[jdMorgan]

From what I understand, Almaden is used to create a knowledge base for sale to IBM clients. As far as I know, it is not publically accessible. Since it costs me bandwidth, provides me with no known benefit and has misbehaved badly in the past, I regret to say that it is still not welcome on my sites. YMMV

Jim

[Wizcrafts]

Any opinion on the Almaden bot mentioned above?

AFAIK, it's a search engine from IBM. I allow it to index my sites. It does no harm, and doesn't follow banned or hidden links.

Wiz

[guillermo5000]

Thank you both!