blank UA and referrer

receiving 403 from SEW's Server Header Checker

[nancyb]

I was getting a lot of blank UAs and referrers in my logs, so searched around WebmasterWorld and found the following:

RewriteCond %{HTTP_USER_AGENT} ^-?$ [OR]

but, I'm not sure the result is what I want.

When I use SEW's Server Header Checker it returns a 403 although the page resolves in the browsers (Opera, IE, and Moz).

I removed that line from my htaccess and the Checker returns a 200 and, of course, the page resolves.

I'm thoroughly confused:

I thought that a 403 would prevent the page from showing in the browser and would return the 403 error file.

There are no denies to my IP address, no denies for the UA shown in the Browser Header Checker:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Opera 7.11 [en]

Ccan someone tell me what I'm evidently too dense to figure out :( Thanks

[Philosopher]

Hmmm...maybe I'm missing the question but...

The rewrite command you are using is designed to block access to visitors that do not send a UserAgent.

This is why the server header checker is being blocked. If memory serves me, it does not send a UA when spidering the target URL.

When a browser such as IE, Opera, Mozilla etc. visits, it will send a UA so it will not be blocked.

[nancyb]

Thanks! Yes, you understood the question, I didn't know that the Checker didn't send a UA.

I started looking around for problems when I discovered that one page of my site consistently returns a 301 for a page that should not and tried the Server Header checker.

So, now I just have to figure out why:

123.456.78.90 - - [26/May/2004:12:20:38 -0400] "GET /subdirectory/index.htm HTTP/1.1" 301 267 "http://www.mydomain.com/file.htm" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/124 (KHTML, like Gecko) Safari/125.1"

123.456.78.90 - - [26/May/2004:12:20:38 -0400] "GET /subdirectory/index.htm HTTP/1.1" 200 18351 "http://www.mydomain.com/file.htm" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/124 (KHTML, like Gecko) Safari/125.1"

I just changed to cable because ADSL service with phone company is so terribly slow. Of course, now I'm running into all kinds of inconsitencies I'm not familiar with. The 301 is not related to this though as I've seen it before and just never got around to checking it out since it always resolves with a 200 on the next GET.

Thanks for your response, Philosopher.

[Philosopher]

Glad I could help.

I would imagine that you don't get the 301 on the second attempt due to some type of a cache issue.

[nancyb]

But, why would it give a 301 in the first place since there is nothing in htaccess that calls for a redirect on this file?

Guess I'll call hosting service and see if they can clear it up. This has been going on since early April for a number of different IPs although some seem to get a 200 instead of the 301. Strange!

Thanks, again!

[py9jmas]

Are you redirecting [example.com...] to [example.com...] Since the access log doesn't record the domain that was requested, this is what I'd expect to see if this was the case.

[nancyb]

Yep, py9jmas. :) I just discovered that was the issue and came back to post it and discovered your message.

Spent 20 min on the phone with tech support before they gave up and asked me to send lines from the log files so they could investigate further.

While putting the email together for them I remembered the www-non-www redirect.

I didn't think of it because I've never been hosted anywhere before where I had to worry about the www and non-www issue. I added the redirect to my htaccess about a month ago when when this "issue" began.

I had these lines in the htaccess first and they caused a 501, if I remember correctly:

RewriteCond %{HTTP_HOST} ^mydomain\.com
RewriteRule (.*) [mydomain.com...] [R=301,L]

I changed the rewrite to the following:

RewriteCond %{HTTP_HOST}!^www\.mydomain\.com
RewriteRule (.*) [mydomain.com...] [R=301,L]

Am I correct that the second example is the correct one?

[jdMorgan]

Either should work if your goal is simply to redirect the non-www to the www- domain. The first one is very specific, and says "redirect non-www to www-", while the second one says "redirect anything that is not www- to www-". The first one would not redirect test.example.com to www.example.com, while the second one would.

It would be quite unusual for the first code sample to cause an error and the second one to work. I'd expect both or neither to work.

Jim

[nancyb]

Thanks, Jim!

I don't have any sub domains so guess I don't need the second example, but after I uploaded the first code I couldn't get to any page on my site.

I just re-checked the http error codes and it was a 500 Server Error, not a 501. Guess if the second works ok, I won't fix what ain't broke ;)

Thanks for the clarification
Nancy