Stopping Google From Crawling Non-existent Pages Via .htaccess

[geckofuel]

Somehow Google is now crawling URLs at my site that do not exist and I can find no links to these pages. All the URLs look like this:

[domain.com...]

While [domain.com...] is a *real* page, none of the tacked on information is. In fact, page.php is not a dynamic page at all. However, the server currently serves up the same content for each of these URLs.

What I want to do is put in a permanent redirect or rewrite that does the following:

If this is requested:
[domain.com...]

Then redirect or rewrite to:
[domain.com...]

This is what I tried but it is not working:

RewriteEngine on
RewriteRule ^page\.php/(.*)$ page\.php [L,R=301]

The basic idea here is that I want to drop all the tacked on URL content after the page.php - Any suggestions?

[jdMorgan]

geckofuel,

There's no problem with what you've posted, but it may be something that's missing or disabled.

How does it not work? Are you not getting a redirect, or are you getting a server error?

If so, what's on your server error log?

Is this code located in .htaccess in your document_root directory or somewhere else?

Do you have any other mod_rewrite rules that do work? (It's possible you're not allowed to use mod_rewrite if you're on an inexpensive shared host.)

Jim

[geckofuel]

jdMorgan,

Sorry I wasn't more clear. It doesn't work because instead of redirecting to page.php, a 404 error page comes up (page does not exist). So, it must not be rewriting to page.php for some reason because I know that page.php is on the server and functioning properly.

Yes, I have several mod_rewrite directives that do work.

[jdMorgan]

Is this code located in .htaccess in your document_root directory or somewhere else?

?

[jd01]

.htaccess version:
RewriteRule ^page\.php/(.*)$ /page.php [L,R=301]

httpd.conf version:
RewriteRule ^/page\.php/(.*)$ page.php [L,R=301]

In .htaccess the preceding / is stripped prior to any comparrison, but should be present in the full path to the file location.

In the httpd.conf file, the preceding / is present for comparrison, but is not necessary in the full path to the file.

The escaping of the meta .(dot) should not be present on the right side of the rule.

I also believe I remember reading that a redirect, which generates an external (visible) response should be the full canonical URL... EG http://yoursite.com/page.php [R=301,L] would be the right-side of the rule.

Jim, please correct me if I am wrong.

Hope this helps.

Justin

[jdMorgan]

The slash on the right side (substitution URL) is independent of whether the code is located in .htaccess or httpd.conf. It depends soley on whether you want the substitution to be constructed from the current local URL-path plus the given substitution, or if you wish to explicitly state the substitution path from Document_Root.

Yes, external redirects should have both the canonical URL and an [R] flag -- and specifically [R=301,L] for almost all cases. This canonical URL can be explicit or constructed by expanding server variables, such as "http://%{HTTP_HOST}/xyz%{REQUEST_URI}".

This is not an absolute requirement, but it seems that many servers have configuration problems beyond the Webmaster's control that result in errors if the non-canonical forms are used. I suspect it's a problem with the ServerName and the UseCanonicalName setting, but since all examples I've seen have been on commercially-hosted sites, I've never been able to confirm it.

Jim