Forum Moderators: phranque

Message Too Old, No Replies

404 issue

Website has many 404 in index.

         

seo sitemaker

5:33 pm on May 17, 2010 (gmt 0)

10+ Year Member



Hello,

This website I am working on had a messed up .htaccess about two months ago, which is supposed to be fixed, but the googlebot keeps on finding lots of 404s, there is over 40,000 of them in webmaster tools. These pages are not in the google index, or sitemap, nor any links from any other pages, as far as we can tell. It seems to be some error in the .htaccess file.

Google is not crawling pages anymore like it used to, pages are being dropped out of google index, traffic has gone down. So I would really like to get it fixed as soon as possible.

This is part of the file:
#Options +FollowSymLinks

php_value error_reporting 1

<IfModule mod_rewrite.c>
RewriteEngine On

RewriteCond %{HTTPS} !on
RewriteCond %{HTTP_HOST} ^website.com [NC]
RewriteRule ^(.*)$ http://www.website.com/$1 [L,R=301]

RewriteCond %{HTTPS} on
RewriteCond %{HTTP_HOST} ^website.com [NC]
RewriteRule ^(.*)$ https://www.website.com/$1 [L,R=301]

RewriteRule ^(.*)\/function\.mkdir$ /$1 [L,R=301]

RewriteRule ^google2a182b0273bc.html$ - [L]
RewriteRule ^google493735a2a7cd7.html$ - [L]

RewriteRule (.*)\.html$ $1.php [L]
RewriteRule (.*)sitemap\.xml$ sitemap_area.php [L]

RewriteRule api - [L]

RewriteRule ^(.*)\/$ /$1 [R=301,L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . frontend/index.php [L]

RewriteCond %{REQUEST_URI} !^/robots.txt
RewriteRule ^[A-Z]{2}-.* /php/profile.php [L]
</IfModule>

#RedirectMatch 301 ^/(.*)\/$ /$1


If anyone can help me, please let me know and I can give you more details.

g1smd

9:05 pm on May 17, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The biggest issues are:

You have mixed RewriteRule and Redirect/RedirectMatch directives in the same file.

Use RewriteRule for all of the rules.

Always include the target domain name in the rule when you have an external redirect.

You have external redirects listed after the internal rewrites. List all external redirects in order of most specific to least specific first. After that list all the internal rewrites, again in order of most to least specific.

Some rules never get to be processed because a prior rule incorrectly grabs all requests. Fix the types and the order and we'll take another look.

seo sitemaker

1:51 pm on May 18, 2010 (gmt 0)

10+ Year Member



You have mixed RewriteRule and Redirect/RedirectMatch directives in the same file.
Use RewriteRule for all of the rules.

You can't have both? Some files are being rewritten as a part of the cms, and some are being redirected because the links come from other sites and the pages don't exist anymore.

Always include the target domain name in the rule when you have an external redirect.
You have external redirects listed after the internal rewrites.

Which external redirects?

List all external redirects in order of most specific to least specific first. After that list all the internal rewrites, again in order of most to least specific.

A little more help on this, not sure I get this. Newbie:)

Some rules never get to be processed because a prior rule incorrectly grabs all requests. Fix the types and the order and we'll take another look.

Like where, could I have an example of this?

I am not familiar at all with this stuff, I appreciate any help.

jdMorgan

5:26 pm on May 18, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You have mixed RewriteRule and Redirect/RedirectMatch directives in the same file.
Use RewriteRule for all of the rules.

You can't have both? Some files are being rewritten as a part of the cms, and some are being redirected because the links come from other sites and the pages don't exist anymore.

No, you can't have both if you expect to control the order of directive execution. On some servers, mod_alias directives will be processed before mod_rewrite, and on others, the reverse is true. If you change hosts, upgrade your server, or modify a config file, you could be faced with a huge list of redirects and rewrites that no longer work as expected...

Always include the target domain name in the rule when you have an external redirect.
You have external redirects listed after the internal rewrites.

Which external redirects?

Well, all of them. Any Redirect or RedirectMatch, and any RewriteRule that ends with [R=301,L] should specify a full canonical URL including protocol, hostname, URL-path, and (if needed) query string.

List all external redirects in order of most specific to least specific first. After that list all the internal rewrites, again in order of most to least specific.

A little more help on this, not sure I get this. Newbie:)

Analyze the patterns in your rules --within the two external redirect and internal rewrite groups that g1smd specified-- and put them in order from most-specific patterns and conditions to least-specific patterns and conditions. Those which are most-specific affect one or only a few possible URL-requests. Those which are least-specific will affect many, many possibly-requested URLs.

Some rules never get to be processed because a prior rule incorrectly grabs all requests. Fix the types and the order and we'll take another look.

Like where, could I have an example of this?

 RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

Redirects all requests for *any* URL-paths with a non-blank non-canonical hostname to the same URL-paths on the canonical host. It is therefore one of the "least-specific" rules. If you were to put that rule *above* this very-specific rule:
RewriteRule ^foo\.html$ http://www.example.com/bar.html [R=301,L]

and a request then arrived at your server for "example.com/foo.html" --note no "www" in that hostname-- then the result would be two redirects in a row -- the first to www.example.com/foo.html correcting the hostname, and then the second to www.example.com/bar.html redirecting to the new URL-path.

That's not what you'd want. Reverse the order of the two rules, and the result is a single redirect to the correct URL-path on the correct domain. The now-second rule now serves only as a "catch-all at the end" for non-canonical requests which otherwise need not be redirected to specific replacement URLs.

I am not familiar at all with this stuff, I appreciate any help.

Perhaps the best help would be the admonishment that none of this is simple, and the potential exists for great damage to the correct operation and search rankings of your site. Success will require intense research, and you cannot depend on 'some guy on some forum' who knows nothing about your site's architecture or operation for this kind of thing, regardless of your schedule pressure or that forum guy's perceived expertise. It's likely a choice between getting it done correctly but slowly, or getting it done incorrectly and fast. This is server configuration code, and there are no short-cuts... :(

The resources cited in our Apache Forum Charter, and the example and tutorial threads in our Apache Forum Library may be helpful in this regard.

Jim

seo sitemaker

8:32 pm on May 18, 2010 (gmt 0)

10+ Year Member



Perhaps the best help would be the admonishment that none of this is simple, and the potential exists for great damage to the correct operation and search rankings of your site. Success will require intense research, and you cannot depend on 'some guy on some forum' who knows nothing about your site's architecture or operation for this kind of thing, regardless of your schedule pressure or that forum guy's perceived expertise. It's likely a choice between getting it done correctly but slowly, or getting it done incorrectly and fast.

Alirght you convinced me:) So where do I go to get it done correctly?

jdMorgan

9:54 pm on May 18, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This was the main point:

> The resources cited in our Apache Forum Charter, and the example and tutorial threads in our Apache Forum Library may be helpful in this regard.

Jim

seo sitemaker

1:15 pm on Jun 16, 2010 (gmt 0)

10+ Year Member



Suppose I were to ask someone for help, where would be a good place to find an honest expert?