Google Network Unreachable Help

traffic is way down

[vfetty]

I have a huge issue here, google says I have over 44,000 urls that are Network Unreachable. I switched from ASP solution forum to PHP and now my traffic is 70% down. I must have done something wrong with my 301 redirects but cannot figure it out. Can anyone have a look please!

.htaccess file

<IfModule mod_rewrite.c>
Options +FollowSymLinks
Options +Indexes
RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} TOPIC_ID=([0-9]+)
RewriteRule ^topic.asp$ /forum/index.php?showtopic=%1 [R=301,L,QSA]
RewriteCond %{QUERY_STRING} FORUM_ID=([0-9]+)
RewriteRule ^forum.asp$ /forum/index.php [R=301,L,QSA]
RewriteCond %{QUERY_STRING} ^TOPIC_ID=([0-9]+)&whichpage=([0-9]+)
RewriteRule ^forum.asp$ /forum/index.php [R=301,L,QSA]
RewriteCond %{QUERY_STRING} whichpage=([0-9]+)
RewriteRule ^forum.asp$ /forum/index.php [R=301,L,QSA]
RewriteCond %{QUERY_STRING} ^showtopic\=213083\&TOPIC_ID\=([^&]+)$
RewriteRule ^$ /forum/index.php? [R=301,L]
RewriteRule ^forum.asp$ /forum/index.php [R=301,L,QSA]
</IfModule>

I ran a headers tool and this is what I got back.

1. Requesting: http://example.com/forum
HEAD /forum HTTP/1.1
Connection: Keep-Alive
Keep-Alive: 300
Accept:*/*
Host: example.com
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Server Response: 301 Moved Permanently
Date: Sun, 06 Jan 2008 19:52:24 GMT
Server: Apache/2.2.3 (Red Hat)
Location: http://example.com/forum/
Connection: close
Content-Type: text/html; charset=iso-8859-1

Redirecting to http://example.com/forum/ ...

2. Requesting: http://example.com/forum/
HEAD /forum/ HTTP/1.1
Connection: Keep-Alive
Keep-Alive: 300
Accept:*/*
Host: example.com
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Server Response: 500 Internal Server Error
Date: Sun, 06 Jan 2008 19:52:24 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.2.4
Connection: close
Content-Type: text/html

This is an example of the old URL it is trying to spider and I was trying to 301 redirect.

http://www.example.com/forum/forum.asp?FORUM_ID=2&whichpage=10

Thanks for any help here!

[jdMorgan]

You should not use [QSA] when trying to 'get rid of' query strings. If you do, you may create an infinite redirection loop because the query string won't be removed, and will appear appended to the redirected URL in the subsequent client request.

You must also explicitly clear the query string by appending a "?" to the substitution URL -- again to prevent redirection loops.

Finally, your rules are not in the correct order, which may also cause problems. Put the most-specific rules first.

Options +FollowSymLinks +Indexes
RewriteEngine on
#
RewriteCond %{QUERY_STRING} ^showtopic\=213083\&TOPIC_ID\=([^&]+)$
RewriteRule ^$ /forum/index.php? [R=301,L]
#
RewriteCond %{QUERY_STRING} TOPIC_ID=([0-9]+)
RewriteRule ^topic.asp$ /forum/index.php?showtopic=%1 [R=301,L]
#
RewriteCond %{QUERY_STRING} ^TOPIC_ID=([0-9]+)&whichpage=([0-9]+)
RewriteRule ^forum.asp$ /forum/index.php? [R=301,L]
#
RewriteCond %{QUERY_STRING} FORUM_ID=([0-9]+)
RewriteRule ^forum.asp$ /forum/index.php? [R=301,L]
#
RewriteCond %{QUERY_STRING} whichpage=([0-9]+)
RewriteRule ^forum.asp$ /forum/index.php? [R=301,L]
#
RewriteRule ^forum.asp$ /forum/index.php [R=301,L,QSA]

This may not fix all the problems you have. In fact, it probably won't. But whenever you get a 500-Server Error, your first stop should be at the server error log file -- It will often tell you exactly what is wrong.

Jim

[vfetty]

hey jdMorgan, Thanks a bunch for helping me out with that htaccess stuff. Also when I do a googlebot spider check via web tool it says this Invalid URL or server does not respond, HTTP return code: 301, does this mean google will still not spider the urls?

[vfetty]

Also forgot to add it's giving me this

Meta Tag Analyzer
Error: 500 Internal Server Error

[jdMorgan]

But whenever you get a 500-Server Error, your first stop should be at the server error log file -- It will often tell you exactly what is wrong.

Read your error log file.

Jim

[edited by: jdMorgan at 3:27 am (utc) on Jan. 7, 2008]

[vfetty]

Thanks but I did and it shows me the exact same thing, 500 error for googlebot + the header type. So that is why I am confused when human eyes see the page fine and googlebot sees a 500.

[jdMorgan]

If you look at your error log --not your access log-- it is most likely you will see something like this:

[Sun Jan 6 22:21:37 2008] [error] [client 64.***.12.34] mod_rewrite: Maximum number of internal redirects reached. Assuming configuration error. Use 'RewriteOptions MaxRedirects' to increase the limit if necessary.

In this case, the 'correct page' may be returned to the client, but with a 500 status.

You may get a different error message, but it is unmistakably an error message, and as shown, often gives some idea of both the exact cause of the problem and a suggested fix (although not a very good one in this case).

If you are not seeing an explicit [error] message, then you're not looking at the error log file, and without that information, we can be of very little help here.

Jim

[vfetty]

Sorry about that I was looking at the access logs. The error logs are gigs so it has taken me a while to figure out that Ultra Edit 32 was a good fit for reading them. Anyway trying to locate anything to do with mod_rewrite and nothing is coming up. There seems to be a lot of traffic pouring into my 404.php custom page from the /forum link but why? Even still the header analysis tools are saying 301 redirect even when I take out the .htaccess file, I'm baffled. Still do not see anything about the 500 internal error in the error logs, so why in the google webmaster tools is it giving me about 40,000 500 internal errors. I guess at this point I can look at tomorrows logs since these contain nothing that were mentioned. Please let me know if anyone has seen this before. Thanks!