Homepage missing from SERPs - is there a .301 error in my .htaccess?

[JackR]

Today my homepage vanished from Google and despite checking the http headers (correct reponses), I've read a few horror stories about redirect loops and homepages being dropped, etc.

www.example.com returns 200

www.example.com/index.html return 301 to www.example.com


Does this look correct?

Options +FollowSymLinks
RewriteEngine On

# Redirect index.htm/.html to '/'
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html?(\?[^\ ]+)?\ HTTP/ [NC]
RewriteRule ^(([^/]+/)*)index\.html?$ http://www.example.com/$1? [R=301,L]

# If not already redirected above, redirect non-canonical domain requests to the canonical domain
RewriteCond %{HTTP_HOST} !^www\.example\.com$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

# HTTP/1.1 301 Moved Permanently
RewriteRule ^widget.html|big-widgets|even-bigger-widgets|my-widgets/widget\.html? http://www.example.com/widgets-guide.html [R=301,L]

# HTTP/1.1 410 Gone
RewriteRule ^/folder/widgets\.html - [G]

[JackR]

There is indeed an error:

http://www.example.com//

How do I fix the two slashes problem?

[g1smd]

Rules 4 and 3 should be rules 1 and 2 in that order.

Rules 1 and 2 should be rules 3 and 4 in that order.

[JackR]

Thank you!

I corrected the order, but the domain still resolves with www.example.com//


If I add

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

# Redirect to remove multiple slashes before URL-path
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ //+([^\ ]*)
RewriteRule .* http://www.example.com/%1 [R=301,L]

It fixes the two slashes problem .. but I'm guessing that's still not the correct way to do it?

[JackR]

It seems there may be multiple errors in the .htaccess, despite my asking for help and rewriting it substantially.

I really need to get this fixed, so I'm hoping that someone can please tell me what is wrong here:

Options +FollowSymLinks
RewriteEngine On

# HTTP/1.1 301 Moved Permanently
RewriteRule ^blonde.html|blonde-escorts|blonde-escort-london|london-escorts/blonde\.html? http://www.example.com/blonde-london-escorts.html [R=301,L]
RewriteRule ^brunette.html|brunette-escort-london|london-escorts/brunette\.html? http://www.example.com/brunette-london-escorts.html [R=301,L]
RewriteRule ^busty.html|busty-escorts|busty-escort-london|london-escorts/busty\.html? http://www.example.com/busty-london-escorts.html [R=301,L]
RewriteRule ^latina.html|latin-escort-london|brazilian-escort-london|london-escorts/latina\.html? http://www.example.com/latina-london-escorts.html [R=301,L]
RewriteRule ^reviews\.html? http://www.example.com/feedback.html [R=301,L]

# HTTP/1.1 410 Gone
RewriteRule ^Description|TITLE|London-Escorts-Portfolio|London_escorts_portfolio|london-escort-in-chelsea-milla|london-escort-in-paddintgon-angelina|london-escorts/aged-18-21|london-escorts/aged-22-plus|london-escorts/escorts-az\.html - [G]
RewriteRule ^links/independent-escorts-in-london|links/escort-agencies-in-london|links/linking|links/fetish|links/index|links/escort-web-site-design|links/escort-agencies-in-the-usa|links/erotic-stories\.html - [G]
RewriteRule ^banners/ - [G]

# Redirect index.htm/.html to '/'
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html?(\?[^\ ]+)?\ HTTP/ [NC]
RewriteRule ^(([^/]+/)*)index\.html?$ http://www.example.com/$1? [R=301,L]

# If not already redirected above, redirect non-canonical domain requests to the canonical domain
RewriteCond %{HTTP_HOST} !^www\.example\.com$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

# Redirect to remove multiple slashes before URL-path
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ //+([^\ ]*)
RewriteRule .* http://www.example.com/%1 [R=301,L]

Thanks

[lucy24]

Y'know, something tells me I would not care for your site :o

If I add

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

# Redirect to remove multiple slashes before URL-path
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ //+([^\ ]*)
RewriteRule .* http://www.example.com/%1 [R=301,L]

It fixes the two slashes problem .. but I'm guessing that's still not the correct way to do it?

Where are the duplicate slashes coming from? If they're from outside you can't do anything about them; if they're internal you need to fine-tooth-comb the site. You can set the Link Checker to several layers of recursion-- enough to hit your whole site in one sweep-- and go have dinner. Or go to sleep, depending on how big the site is.

Your final rule is unnecessary because it is included within the previous rule:

RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L] 

If the value of the first * is zero, that means the request begins with a slash.

Spaces \ are scary. If your htaccess will recognize the \S form, use it.

You never need an unanchored and uncaptured .* You can stop at .?

RewriteRule ^/folder/widgets\.html - [G] 

This will always fail, because mod_rewrite doesn't use leading slashes. Unless, ahem, you have two of them in a row.

RewriteRule ^widget.html|big-widgets|even-bigger-widgets|my-widgets/widget\.html? http://www.example.com/widgets-guide.html [R=301,L]

Whoops! Come on, you know better. This is equivalent to

^widget.html[OR]
big-widgets[OR]
even-bigger-widgets[OR]
my-widgets/widget\.html?

I kinda doubt that was what you intended. You need a set of parentheses. Here they're not for capturing, just for grouping the pipe-alternations.

[JackR]

That is indeed a fair point Lucy, so thank you for taking the time to reply.

The file below was largely based on my misunderstanding of the help given by yourself and g1smd in previous threads.

The problem is that I simply don't know where the '//' is coming from. It's currently not referred to in any single .html file or elsewhere as far as I can tell.


Here's my next attempt:

Options +FollowSymLinks
RewriteEngine On

# HTTP/1.1 301 Moved Permanently
RewriteRule blonde.html|blonde-escorts|blonde-escort-london|london-escorts/blonde\.html? http://www.example.com/blonde-london-escorts.html [R=301,L]
RewriteRule brunette.html|brunette-escort-london|london-escorts/brunette\.html? http://www.example.com/brunette-london-escorts.html [R=301,L]
RewriteRule busty.html|busty-escorts|busty-escort-london|london-escorts/busty\.html? http://www.example.com/busty-london-escorts.html [R=301,L]
RewriteRule latina.html|latin-escort-london|brazilian-escort-london|london-escorts/latina\.html? http://www.example.com/latina-london-escorts.html [R=301,L]
RewriteRule reviews\.html? http://www.example.com/feedback.html [R=301,L]

# HTTP/1.1 410 Gone
RewriteRule Description|TITLE|London-Escorts-Portfolio|London_escorts_portfolio|london-escort-in-chelsea-milla|london-escort-in-paddintgon-angelina|london-escorts/aged-18-21|london-escorts/aged-22-plus|london-escorts/escorts-az\.html - [G]
RewriteRule links/independent-escorts-in-london|links/escort-agencies-in-london|links/linking|links/fetish|links/index|links/escort-web-site-design|links/escort-agencies-in-the-usa|links/erotic-stories\.html - [G]

# Redirect index.htm/.html to '/'
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html?(\?[^\ ]+)?\ HTTP/ [NC]
RewriteRule ^(([^/]+/)*)index\.html?$ http://www.example.com/$1? [R=301,L]

# If not already redirected above, redirect non-canonical domain requests to the canonical domain
RewriteCond %{HTTP_HOST} !^www\.example\.com$
RewriteRule ^(.?)$ http://www.example.com/$1 [R=301,L]

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.?)$ http://www.example.com/$1$3 [R=301,L]

I'm sure it's still fail but I'm determined to get there with some guidance.

[JackR]

Every rewrite, etc appears to be working with this (hopefully) final code!

Any improvements you folks can suggest?

Options +FollowSymLinks
RewriteEngine On

# HTTP/1.1 301 Moved Permanently
RewriteRule (blonde|blonde-escorts|blonde-escort-london|london-escorts/blonde)\.html? http://www.example.com/blonde-london-escorts.html [R=301,L]
RewriteRule (brunette|brunette-escort-london|london-escorts/brunette)\.html? http://www.example.com/brunette-london-escorts.html [R=301,L]
RewriteRule (busty|busty-escorts|busty-escort-london|london-escorts/busty)\.html? http://www.example.com/busty-london-escorts.html [R=301,L]
RewriteRule (latina|latin-escort-london|brazilian-escort-london|london-escorts/latina)\.html? http://www.example.com/latina-london-escorts.html [R=301,L]
RewriteRule ^reviews\.html? http://www.example.com/feedback.html [R=301,L]

# HTTP/1.1 410 Gone
RewriteRule (London-Escorts-Portfolio|Description|TITLE|banners|links/independent-escorts-in-london|links/escort-agencies-in-london|links/linking|links/fetish|links/index|links/escort-web-site-design|links/escort-agencies-in-the-usa|links/erotic-stories|links/linkingThanks|links/London-Escort-Agencies|London_escorts_portfolio|london-escort-in-chelsea-milla|london-escort-in-paddintgon-angelina|london-escorts/aged-18-21|london-escorts/aged-22-plus|london-escorts/escorts-az|london-escort-in-bayswat..|london-esco..|&usg=ALkJrhgz6MRDcFkp-kcCoKgNS9ERG-CLtQ|$myEmail|London_escorts_portfolio.htm?a=g|zoe.html?iframe=true&width=80%&height=80%)\.html? - [G]

# Return 410-Gone for myEmail URLs
RewriteRule myEmail - [G]

# Redirect index.htm/.html to '/'
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html?(\?[^\ ]+)?\ HTTP/ [NC]
RewriteRule ^(([^/]+/)*)index\.html?$ http://www.example.com/$1? [R=301,L]

# If not already redirected above, redirect non-canonical domain requests to the canonical domain
RewriteCond %{HTTP_HOST} !^www\.example\.com$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

# Redirect to remove multiple slashes before URL-path
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ //+([^\ ]*)
RewriteRule .* http://www.example.com/%1 [R=301,L]

# 301 redirect requests to files with no extension to same with appended .html extension
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\..+$
RewriteRule ^(.*)$ /$1.html [R=301,L]

[lucy24]

Edit:

Oops. Didn't realize my last post overlapped yours. You had two in a row of the same size & shape. Oh well.

Go back and put in those parentheses. It is safest to make a separate rule for each directory:

RewriteRule ^(beachboys|musclemen|beefcake|wiseguys)\.html?$ http://www.example.com/basic-package.html [R=301,L]

RewriteRule ^twins/(identical|fraternal|unexpected|inverted)\.html?$ http://www.example.com/twofers.html [R=301,L]

RewriteRule ^gigolos/(american|continental|part-time|professional)\.html?$ http://www.example.com/its-your-money.html [R=301,L]

RewriteRule ^(aspiring-models|unemployed-actors|unpublished-writers)\.html?$ - [G,L]

et cetera.

[JackR]

Aha! Now I see!


I didn't overlook your point:

Your final rule is unnecessary because it is included within the previous rule:

RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

In fact, if I do omit the final rule:

RewriteRule .* http://www.example.com/%1 [R=301,L]

...then http://www.example.com// is a valid URL giving a 200.

With the line in place, the code is a .301. redirect from "//" to "/"

Which is preferable?

After all, [webmasterworld.com...] returns a standard 200 code!

[g1smd]

The rule order is wrong. The non-www to www rule must always be the last redirecting rule.

If it isn't, some requests will see an unwanted double redirect.

[JackR]

Sorry, you're right. This should now be correct - I hope!:

Options +FollowSymLinks
RewriteEngine On

# HTTP/1.1 301 Moved Permanently
RewriteRule (blonde|blonde-escorts|blonde-escort-london|london-escorts/blonde)\.html? http://www.example.com/blonde-london-escorts.html [R=301,L]
RewriteRule (brunette|brunette-escort-london|london-escorts/brunette)\.html? http://www.example.com/brunette-london-escorts.html [R=301,L]
RewriteRule (busty|busty-escorts|busty-escort-london|london-escorts/busty)\.html? http://www.example.com/busty-london-escorts.html [R=301,L]
RewriteRule (latina|latin-escort-london|brazilian-escort-london|london-escorts/latina)\.html? http://www.example.com/latina-london-escorts.html [R=301,L]
RewriteRule ^reviews\.html? http://www.example.com/feedback.html [R=301,L]

# HTTP/1.1 410 Gone
RewriteRule ^links/(independent-escorts-in-london|escort-agencies-in-london|linking|fetish|index|escort-web-site-design|escort-agencies-in-the-usa|erotic-stories|linkingThanks|London-Escort-Agencies)\.html? - [G]
RewriteRule ^london-escorts/(aged-18-21|aged-22-plus|escorts-az)\.html? - [G]
RewriteRule ^(London-Escorts-Portfolio|London_escorts_portfolio|Description|TITLE|banners|london-escort-in-chelsea-milla|london-escort-in-paddintgon-angelina|london-escort-in-bayswat..|london-esco..|&usg=ALkJrhgz6MRDcFkp-kcCoKgNS9ERG-CLtQ|$myEmail|London_escorts_portfolio.htm?a=g|zoe.html?iframe=true&width=80%&height=80%)\.html? - [G]

# Return 410-Gone for myEmail URLs
RewriteRule myEmail - [G]

# Redirect index.htm/.html to '/'
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html?(\?[^\ ]+)?\ HTTP/ [NC]
RewriteRule ^(([^/]+/)*)index\.html?$ http://www.example.com/$1? [R=301,L]

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

# Redirect to remove multiple slashes before URL-path
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ //+([^\ ]*)
RewriteRule .* http://www.example.com/%1 [R=301,L]

# 301 redirect requests to files with no extension to same with appended .html extension
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\..+$
RewriteRule ^(.*)$ /$1.html [R=301,L]

# If not already redirected above, redirect non-canonical domain requests to the canonical domain
RewriteCond %{HTTP_HOST} !^www\.example\.com$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

[g1smd]

I always list rules that lead to [G] or [F] before any that redirect to another URL.

No point redirecting a request only to then block it with a later rule.

[JackR]

Thank you - that makes perfect sense in fact.

I've revised the order and have one more question: can you see any issue whatsoever which could lead to duplicate content using the final .htaccess below?

(all/any redirections at the server level are disabled, leaving everything to this .htaccess.)

Options +FollowSymLinks
RewriteEngine On

# HTTP/1.1 410 Gone
RewriteRule ^links/(independent-escorts-in-london|escort-agencies-in-london|linking|fetish|index|escort-web-site-design|escort-agencies-in-the-usa|erotic-stories|linkingThanks|London-Escort-Agencies)\.html? - [G]
RewriteRule ^london-escorts/(aged-18-21|aged-22-plus|escorts-az)\.html? - [G]
RewriteRule ^(London-Escorts-Portfolio|London_escorts_portfolio|Description|TITLE|banners|london-escort-in-chelsea-milla|london-escort-in-paddintgon-angelina|london-escort-in-bayswat..|london-esco..|&usg=ALkJrhgz6MRDcFkp-kcCoKgNS9ERG-CLtQ|$myEmail|London_escorts_portfolio.htm?a=g|zoe.html?iframe=true&width=80%&height=80%)\.html? - [G]

# Return 410-Gone for myEmail URLs
RewriteRule myEmail - [G]

# HTTP/1.1 301 Moved Permanently
RewriteRule (blonde|blonde-escorts|blonde-escort-london|london-escorts/blonde)\.html? http://www.example.com/blonde-london-escorts.html [R=301,L]
RewriteRule (brunette|brunette-escort-london|london-escorts/brunette)\.html? http://www.example.com/brunette-london-escorts.html [R=301,L]
RewriteRule (busty|busty-escorts|busty-escort-london|london-escorts/busty)\.html? http://www.example.com/busty-london-escorts.html [R=301,L]
RewriteRule (latina|latin-escort-london|brazilian-escort-london|london-escorts/latina)\.html? http://www.example.com/latina-london-escorts.html [R=301,L]
RewriteRule ^reviews\.html? http://www.example.com/feedback.html [R=301,L]

# Redirect index.htm/.html to '/'
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html?(\?[^\ ]+)?\ HTTP/ [NC]
RewriteRule ^(([^/]+/)*)index\.html?$ http://www.example.com/$1? [R=301,L]

# Redirect to remove multiple slashes within URL-path
RewriteRule ^(([^/]+/)*)/+(.*)$ http://www.example.com/$1$3 [R=301,L]

# Redirect to remove multiple slashes before URL-path
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ //+([^\ ]*)
RewriteRule .* http://www.example.com/%1 [R=301,L]

# 301 redirect requests to files with no extension to same with appended .html extension
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\..+$
RewriteRule ^(.*)$ /$1.html [R=301,L]

# If not already redirected above, redirect non-canonical domain requests to the canonical domain
RewriteCond %{HTTP_HOST} !^www\.example\.com$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

[g1smd]

Every redirect must contain the domain name in the target otherwise a non-www request might pass through an unwanted redirection chain.

[JackR]

As far as I am aware, every redirect above does contain the domain name in the target.

Or did I miss one?

[g1smd]

Yes, one has been missed.

[JackR]

I'm slowly growing to appreciate your cryptic replies g1smd!

You could at least indicate which line contains the error .. I did paste this htaccess before and got the all clear!

It could well be that my site has been removed from the Google index due to this issue.

That said, is the missing rule a likely candidate for having effectively ruined my sites hard-earned trust?

[g1smd]

Redirects are those with the [R] flag.

The domain name shows as http://www.example.com/ in the code.

Look for a RewriteRule where you have [R] but not http://www.example.com/ present.

[JackR]

Ok, thank you. The only [R] flag with a missing URL is the one below which appends a .html extension where no other extension is present.

Or what am I missing?

# 301 redirect requests to files with no extension to same with appended .html extension
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !\..+$
RewriteRule ^(.*)$ /$1.html [R=301,L]

[lucy24]

Looks like that's the one. Appending the full protocol-and-domain also robs your server of any chance to pull some funny business by putting the wrong number of slashes between the domain name and the rest of the path ;)

This Condition seems awfully generic:

RewriteCond %{REQUEST_URI} !\..+$

It doesn't just exclude requests ending in period-plus-extension, which I assume is what you intended. It excludes any request that contains a period anywhere. These aren't really supposed to occur, but it might be safer to say !\.\w+$ instead.

Come to think of it, wouldn't this go better in the Rule itself? Then the server doesn't have to go back and evaluate the Condition for every single request the way it would with the ultra-generic (.*).

RewriteRule ^([^.]+)$ http://www.example.com/$1.html [R=301,L]

It has to be + rather than * because otherwise you might come out with

http://www.example.com/.html

[g1smd]

Without the domain name you have an unwanted multiple step redirection chain:

example.com/wibble => example.com/wibble.htm => www.example.com/wibble.htm

[JackR]

Thanks Lucy!

RewriteRule ^([^.]+)$ http://www.example.com/$1.html [R=301,L]

Now correctly added to the .htaccess :)

Can you see any way in which the fail version -might- have harmed the site? (it was only in the .htaccess for around 5 days).


EDIT: DAMN! I can't believe I made such a stupid mistake before clarifying 100% that the .htaccess was valid and correct.

So I really did my site some damage as far as Google is concerned.

I'm gutted. :(

[g1smd]

Google has always said you lose a "bit of PR" though a redirect, but retain most and "PR is unlikely to flow through a chain of redirects".

Recently they have said that "Google is now able to follow a chain of redirects, doesn't give up so easily".

Don't beat yourself up on this type of error. It's not a biggie, and it is now fixed.