generate 410 Gone errors?

[nuebeedoo]

I recently needed to move a site to a new URL.

The Permanent Redirects worked great in preserving my SE rankings.

However, there are still way too many external links back to the OLD URL. Basically, people are not updating their sites and bookmarks because the permanent redirect takes them where they want to go.

I then changed the redirect to the error page, forcing them to choose the home or sitemap. But that hasn't cut back on the use of the old URL.

Now, would the next step be to generate a:
410 Gone, and what would be the syntax for that?

My current htaccess contains:
RedirectPermanent / h*tp://www.newURL.tld/errors/moved.html

Thanks!

[jdMorgan]

nuebeedoo,

Welcome to WebmasterWorld [webmasterworld.com]!

This thread [webmasterworld.com] may be useful to you.

Jim

[nuebeedoo]

thanks jim -

I actually did read that thread prior, but mod_rewrite is NOT a skill of mine by any means ...

So I'm assuming there is no Mod_alias for a 410?
I could not locate htaccess info at the w3-org link or at apache-org.

If I try the mod_rewrite, I just want ANY request to the URL root and all pages, folders, files etc to be told GONE.

would that just be?:

RewriteEngine on
RewriteCond %{REQUEST_URI} ^/
RewriteRule .* - [G]

- kathy

[jdMorgan]

kathy,

Well, yes and no. You can't return a 410-Gone response to an HTTP/1.0 request, so that was the thrust of that previous post. The idea is, use the RewriteCond that tests for http version, and if HTTP/1.1 or above, return 410-Gone. If not, return the default 404 not found. 410-Gone is meaningless to an HTTP/1.0 client; it just means "error". Only clients at and above HTTP/1.1 can interpret it as meaning "Gone."

So, I'd recomend that you include the version check.

Jim

[nuebeedoo]

sorry to be so dense but ...
my htaccess looks like this now:

RewriteEngine on

# Respond with 410-Gone status to HTTP/1.1+ requests for removed resources.
# HTTP/1.0 requests will "fall through" and invoke default server 404 handling.
RewriteCond %{THE_REQUEST} ^[^\ ]+\ [^\ ]+\ HTTP/(1\.[1-9]¦[2-9]\.[0-9])
RewriteCond %{REQUEST_URI} ^/
RewriteRule .* - [G]

and all I get are "Forbidden" errors ...
again, I may have edited wrong, as the post you refer to was about specific html pages that are gone, I want all to serve up gone ... so is
RewriteCond %{REQUEST_URI} ^/
correct or am I missing more code ...

- kathy

[jdMorgan]

kathy,

You don't need to test REQUEST_URI at all, since you want to reject all requests:

# Respond with 410-Gone status to HTTP/1.1+ requests for removed resources.
# HTTP/1.0 requests will "fall through" and invoke default server 404 handling.
RewriteCond %{THE_REQUEST} ^[^\ ]+\ [^\ ]+\ HTTP/(1\.[1-9]¦[2-9]\.[0-9])
RewriteRule .* - [G]

Additionally, you may wish to create two custom error pages, in the style of your site. You don't have to, but you could then place a message on each of them asking webmasters and visitors to please update their links and bookmarks. To invoke theses custom pages, simply add:

ErrorDocument 404 /yourcustom404page.html
ErrorDocument 410 /yourcustom410page.html

to your file, ahead or behind the rewriterule stuff.

In order to serve those pages, you'll have to exclude them in the RewriteRule:

RewriteRule !^(yourcustom404page¦yourcustom410page)\.html - [G]

Of course, you can also have custom pages for 403-Forbidden and 500-Server Error as well.

However, this does not explain why you would get 403-Forbidden; If there was a syntax error in the code, I'd expect a 500-Server Error response, not a 403-Forbidden. So actually, I'm stumped, and all the verbage above is diversionary. What does your raw error log show when you get this 403 error?

Jim

[nuebeedoo]

Thanks - I think it has to do with ALL the pages being deleted from the site, including the index, and the directory index block kicked in.
I'll play around tonight, but a blank index or something to see if that works...

from the errorlog
Directory index forbidden by rule: /home/virtual/site189/fst/var/www/html/

followed by:
Options FollowSymLinks or SymLinksIfOwnerMatch is off which implies that RewriteRule directive is forbidden: /home/virtual/site189/fst/var/www/html

[nuebeedoo]

I put up a blank index. Now the rewrite works, but I thought I put in the error page exclusion, but it's not working: I still get the standard generic error page for 410, with the added line:
The requested resource ****xx
is no longer available on this server and there is no forwarding address. Please remove all references to this resource.
Additionally, a 410 Gone error was encountered while trying to use an ErrorDocument to handle the request.

This is the htaccess:
Options +FollowSymLinks

ErrorDocument 404 /sitemoved404.html
ErrorDocument 410 /sitemoved410.html
ErrorDocument 403 /sitemoved403.html
ErrorDocument 500 /sitemoved500.html

RewriteEngine on

# Respond with 410-Gone status to HTTP/1.1+ requests for removed resources.
# HTTP/1.0 requests will "fall through" and invoke default server 404 handling.
RewriteCond %{THE_REQUEST} ^[^\ ]+\ [^\ ]+\ HTTP/(1\.[1-9]¦[2-9]\.[0-9])
RewriteRule!^(sitemoved404¦sitemoved410¦sitemoved403¦sitemoved500)\.html - [G]
RewriteRule .* - [G]

(I did use the correct solid pipes in the actuall htaccess file)

thanks - almost there!

[jdMorgan]

kathy,

Right there is the problem:

> Options FollowSymLinks or SymLinksIfOwnerMatch is off which implies that RewriteRule directive is forbidden:

You'll need to add:

 Options +FollowSymLinks 

just ahead of your RewriteEngine on directive. Without it, mod_rewrite is disabled on your server.

Jim

[nuebeedoo]

the Options +FollowSymLinks was at the top of the htaccess file already, and AFTER adding the blank "index.html" , the 410 worked 1/2 way ... meaning I got the correct 410 error for the page request, but unable to get the custom error page to display. The generic page says that an additional 410 error was encountered "while trying to use an ErrorDocument to handle the request."
So it seems to me the exclution of the error docs from the 410 re-write is not working.

From access log now:
[22/Jan/2004:11:06:23 -0600] "GET / HTTP/1.1" 410 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
No "error" in for the failed custom page in the error log

So this line:
RewriteRule!^(sitemoved404¦sitemoved410¦sitemoved403¦sitemoved500)\.html - [G]

Is that EXCLUDING the listed pages from the rewrite, or do I have the code wrong? Does the order it appears in the rules list make the difference ... that " .*" rule may be overwriting it?

thanks

[nuebeedoo]

Fixed it and works!
Switched this code order:

RewriteRule!^(sitemoved404¦sitemoved410¦sitemoved403¦sitemoved500)\.html - [G] 
RewriteRule .* - [G] 

For this order

RewriteRule .* - [G] 
RewriteRule!^(sitemoved404¦sitemoved410¦sitemoved403¦sitemoved500)\.html - [G] 

Whew!

[jdMorgan]

kathy,

Well this is confusing -- You should not be using the first line -- RewriteRule .* - [G] -- at all. It should be returning a "gone" response to *all* requests, including those for the custom error pages. In other words, with the first rule in place, the second rule will never be executed, so it doesn't do anything. But the first one should be interfering with the delivery of your custom error pages.

Rolling everything posted above together, something like this should work:

ErrorDocument 403 /sitemoved403.html
ErrorDocument 404 /sitemoved404.html
ErrorDocument 410 /sitemoved410.html
ErrorDocument 500 /sitemoved500.html
 
Options +FollowSymLinks
RewriteEngine on
# Respond with 410-Gone status to HTTP/1.1+ requests for removed resources.
# HTTP/1.0 requests will "fall through" and invoke default server 404 handling.
RewriteCond %{THE_REQUEST} ^[^\ ]+\ [^\ ]+\ HTTP/(1\.[1-9]¦[2-9]\.[0-9])
RewriteRule !^sitemoved(40[34]¦410¦500)\.html - [G]
# - the end -

If not, I sure don't know why -- maybe a strange server configuration, or something outside the current scope of discussion, such as cgi scripting.

(As always, replace the broken pipe "¦" characters above with solid pipe characters from your keyboard before use.)

Jim