googlebot being blocked and other strange issues

[mihomes]

So I get the following in an email :

Time:  Sun Oct 21 10:31:52 2012 -0400
IP:  66.249.73.70 (US/United States/crawl-66-249-73-70.googlebot.com)
Failures: 5 (mod_security)
Interval: 300 seconds
Blocked: Permanent Block

Log entries:

[Sun Oct 21 10:28:28 2012] [error] [client 66.249.73.70] ModSecurity: Access denied with code 501 (phase 2). Match of "rx ^((?:(?:POS|GE)T|OPTIONS|HEAD))$" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "38"] [id "960032"] [msg "Method is not allowed by policy"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [hostname "example.com"] [uri "/"] [unique_id "UIQGjGB-guIAAAwRPyQAAAAB"]

Now, I have CSF installed so after this happens a few times it does a perm ip block. So, it perm blocks googlebot until I manually remove that block.

I looked into the error log of apache and it is always something like so :

[Fri Oct 19 03:34:57 2012] [error] [client 66.249.73.70] ModSecurity: Access denied with code 501 (phase 2). Match of "rx ^((?:(?:POS|GE)T|OPTIONS|HEAD))$" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "38"] [id "960032"] [msg "Method is not allowed by policy"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [hostname "example.com"] [uri "/"] [unique_id "UIECoWB-guIAAHJ5BZoAAAAD"]

[Fri Oct 19 03:34:57 2012] [error] [client 66.249.73.70] File does not exist: /usr/local/apache/htdocs/501.shtml

I did some testing and have come to the conclusion that this is caused by crawling https locations. I have an ssl installed on a site, however, I no longer use it. All files have been removed other than the htaccess in its root dir.

Some more testing and I found that if I try to view https of any of my sites the mod_security kicks in and I block myself. Also, when I try to view any https location of my sites (any site) the browser gives back a message how it cannot connect or the there was a failure.

So, I have two questions :

1 - Shouldn't this be throwing a regular error page like a 501 instead of this connection error stuff (I have error pages setup for all cases). I actually made a 501.shtml file in /usr/local/apache/htdocs/, and while that error does not show any longer it still does not serve the error page. Let alone, shouldn't a default kick in if its not present... in that folder I have 400,401,403,404,500, and now the newly created 501.

2 - How can I stop mod security from doing this for https. I was able to block myself and I certainly do not want G-bot blocked.

[lucy24]

Match of "rx ^((?:(?:POS|GE)T|OPTIONS|HEAD))$" against "REQUEST_METHOD" required.

Now, I realize Apache is not English, but that sounds as if the googlebot is trying something other than POST, GET, OPTIONS or HEAD. What else is there? Within the googlebot's vocabulary, I mean. In your regular logs, what was it trying to do?

File does not exist: /usr/local/apache/htdocs/501.shtml
...
and now the newly created 501

How newly? Do you mean that you went and created it after Apache went looking for it?

shouldn't a default kick in if its not present

A default did kick in. When you tried it on yourself, you didn't get a blank page did you?

while that error does not show any longer it still does not serve the error page

That was cryptic. Do you mean that it won't show you the error page even if you ask for it by name?

[edited by: incrediBILL at 10:27 pm (utc) on Oct 22, 2012]
[edit reason] disabled smilies [/edit]

[mihomes]

Hi Lucy... I've had some great questions lately haha.

At the moment I have 400,401,403,404,500, and now 501 pages in /usr/local/apache/htdocs/ which should be the default error pages for my server. Would it be a good idea to create pages for all the rest?

Here is, what I think, the problem is. If I go to any of my sites and try to access https of any page I get a connection error from the browser, no 'error page'... this then triggers modsec and then after a few times triggers CSF which perm blocks me.

I cannot remember the default action that should happen, but shouldn't an error page from /usr/local/apache/htdocs/, or the site in question if it has one set, be shown rather than this connection page? This is what is setting off modsec.

Another person mentioned the LF_APACHE_404 setting in CSF, but this has always been disabled and is not the problem.

[phranque]

the browser must make a secure connection before any encrypted web protocol stuff can happen.
in other words, the connection error isn't an HTTP thing it's a SSL/TLS thing.

what clues do you get in the server access log file for the failed requests that correspond to those ModSecurity messages in the server error log file?

[mihomes]

Here are some entries from my own ip address in the access log :

myipaddress - - [22/Oct/2012:13:30:16 -0400] "\x16\x03\x01" 501 2008
myipaddress - - [22/Oct/2012:13:30:16 -0400] "\x16\x03\x01" 501 2008
myipaddress - - [22/Oct/2012:13:30:16 -0400] "\x16\x03" 501 2008
myipaddress - - [22/Oct/2012:13:30:16 -0400] "\x16\x03" 501 2008
myipaddress - - [22/Oct/2012:13:32:10 -0400] "\x16\x03\x01" 501 2008
myipaddress - - [22/Oct/2012:13:32:10 -0400] "\x16\x03\x01" 501 2008
myipaddress - - [22/Oct/2012:13:32:10 -0400] "\x16\x03" 501 2008
myipaddress - - [22/Oct/2012:13:32:10 -0400] "\x16\x03" 501 2008
myipaddress - - [22/Oct/2012:13:32:13 -0400] "\x16\x03\x01" 501 2008
myipaddress - - [22/Oct/2012:13:32:13 -0400] "\x16\x03\x01" 501 2008
myipaddress - - [22/Oct/2012:13:32:24 -0400] "\x16\x03" 501 2008
myipaddress - - [22/Oct/2012:13:32:24 -0400] "\x16\x03" 501 2008

this would have happend when I tried to view an https page when there is no ssl.

Modsec then kicked in and my CSF blocked me because of the modsec triggered x times within y timeframe.

[mihomes]

Codes appear to be 'looking for SSL on a non-ssl enabled port'.

[phranque]

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.2

501 Not Implemented
The server does not support the functionality required to fulfill the request. This is the appropriate response when the server does not recognize the request method and is not capable of supporting it for any resource.

this certainly seems appropriate for the circumstances where the connection is made as there is not a recognized HTTP Request Method in those requests.

[mihomes]

Yes, it does, however, it also triggers mod_security. In CSF (firewall) I have it set so after 5 modsec triggers with x seconds the ip is blocked. With that said, I could disable the block when this is triggered, but then other 'legit' modsec triggers would NOT be blocked.

My thought process was somehow make sure modsec is not triggered for invalid https?

[mihomes]

I am also still interested in the 501 issue... the error is now gone because I created a 501 default page on the server, but before that a 404 was returned because it could not find the 501... now that I say that it makes sense... if an error codes page is not available it still reverts to an error page 404.

Anyways... thoughts on this? Would you just disable the trigger on modsec or look into a custom rule for modsec so invalid https are not triggered?

[mihomes]

Update... I removed the modsec block in csf. Two things have changed :

1 - on my old site which I have ssl installed now going to an invalid page with https properly shows an error page.

2 - on sites without ssl when I try to go to a page, at least in ff, I get :

SSL received a record that exceeded the maximum permissible length.

(Error code: ssl_error_rx_record_too_long)

I still think something is wrong because that error message should be something like 'unable to connect' which happens on other sites I have tried viewing https when there is none.

...because of that is why I think modsec is acting up in the first place.

[phranque]

have you tried another browser?
or tried restarting your browser and clearing cache?
there may be some type of connection information that is outdated.

http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html#1040406

"SSL received a record that exceeded the maximum permissible length."

This generally indicates that the remote peer system has a flawed implementation of SSL, and is violating the SSL specification.

you might want to enable SSL logging or do some sniffing to see what's going on - DebuggingSSLProblems - Httpd Wiki:
http://wiki.apache.org/httpd/DebuggingSSLProblems [wiki.apache.org]

[mihomes]

Ran some more tests and all I need to do is visit any of my sites which DO NOT have ssl (in other words my main shared ip) in https and it will trigger mod_sec with :

[Tue Oct 23 00:47:21 2012] [error] [client 99.30.160.94] ModSecurity: Access denied with code 501 (phase 2). Match of "rx ^((?:(?:POS|GE)T|OPTIONS|HEAD))$" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/modsec2.user.conf"] [line "38"] [id "960032"] [msg "Method is not allowed by policy"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [hostname "exampleserver.com"] [uri "/"] [unique_id "UIYhWWB-guIAAHl3FKcAAAAA"]

It does not matter what browser or anything... this certainly cannot be correct if default mod_sec rules are being triggered for it.

Any ideas?

[phranque]

the problem appears to be that your SSL/TLS connection is successful and the web server can't handle the subsequent request.

you want the user agent to get the "Can't connect..." response so the web server never sees a secure web request.

[mihomes]

Can'r be it as there is only one ssl on the server and it is installed on its own ip for its own site. It appears to be working fine, but I can only assume the same problem will happen with it when I remove the cert(no longer doing anything with the site).

I am more concerned with the sites who do not have ssl. Yes, I want a message about not being able to connect with https as there is none, but it shouldn't be triggering mod_sec... which is why I think the above error that happens is incorrect.

[phranque]

pure and simple - with HTTPS you cannot do anything on the web server until the SSL/TLS handshake is complete and the secure connection is made.
mod_sec knows absolutely nothing about the HTTPS request until after SSL/TLS has done its thing.
if you don't make a secure connection, you don't have a mod_sec problem.

i would shut down port 443 for that IP and stop serving the cert.