strange errors in error log

[helenp]

Hi,
I have a rule in my htaccess that tells the server to treat html as php pages.
This is from my .htaccess
AddType application/x-httpd-php5 .htm .html
RewriteEngine On
RewriteCond %{HTTP_HOST} ^example\.com
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

Been seing several times this in my error log

[Thu Jan 19 16:19:32 2012] [error] [client 176.31.104.41] File does not exist: /home/mysite/public_html/index.php, referer: http://www.example.com/index.php

I have my index.htm with a link to home that is on all pages, and its called index.htm not index.php
So I dont understand this error, it says index.php does not exists wich is true but the referrer is myste/index.php

Also sometimes I see error like this:

[Thu Jan 19 23:34:20 2012] [error] [client 213.205.232.76] File does not exist: /home/mysite/myweb/406.shtml, referer: http://www.google.co.uk/search?q=villas+in+marbella&ie=UTF-8&oe=UTF-8&hl=en-gb&client=safari

If anybody can explain if normal as I have another problem (suposse) bug with the host.
Thanks

[edited by: tedster at 10:38 pm (utc) on Jan 20, 2012]
[edit reason] switch to example.com [/edit]

[helenp]

The host just said that the 406 error is due to mod_security, and as I cant see any reason to block somebody coming from google search, he found that the ip in question is blacklisted in spamhaus, I seen many similiar in the past days. Is this a normal practice? I dont think its correct.

[Samizdata]

I'm a little confused by your description but I will try to help.

What you are reporting is from your error log and seems to be correct if:

* You don't have a file named index.php

* You don't have a file named 406.shtml

If both of the above are true then your access log should show a 404 status for the requests, and seeing "File does not exist" in your error log is perfectly normal.

The question then becomes "who is making the requests, and why?".

In the first case (176.31.104.nn) the request appears to come from a French server farm - it is extremely unlikely to be a human visitor, just another unwanted robot (which may be why mod_security is involved).

The second case is more problematic as it appears to be a mobile device from England that is clicking a Google search result - but they would still see your 404 Not Found page, where you presumably have a link to useful content (your access log will show if they used it).

As far as I can see you are not getting a "406 error" at all.

Whether you ask your host, here or on Devshed, the devil is in the detail.

Hope this helps.

...

[tangor]

On the scale of "how many" is this a common (daily, for example) error? Is it consistent? Does it come only from a particular ip or range of ips?

Malformed requests are fairly common on the web and if the site is returning a 4x error, then, in most cases, it's not that big a problem. If it is hundreds/thousands of times daily then, yes, one needs to look at the site to see if there's something wrong.

[helenp]

I do get it often,
heres another one:

[Fri Jan 20 14:21:13 2012] [error] [client 217.214.227.10] File does not exist: /home/mysite/public_html/406.shtml, referer: http://www.google.se/url?sa=t&rct=j&q=marbella+-barcelona&source=web&cd=7&ved=0CG4QFjAG&url=http%3A%2F%2Fwww.exampel.com%2Fsvenska%2F&ei=udoZT824EISH4gTNw-yGDQ&usg=AFQjCNHRWrRfxWgFdx3po4NxKrlr1QCSOQ&sig2=GYD7nKN6eNd2BlfcOlINPg

If you copy the http....you comes to the swedish page

Edited, before the mod do :):
uups, not permitted to have links, so just changed url to exampel...anyway its a link that exists.

[tangor]

Why are you spending time in your ERROR log? Those entries have already been handled by the server and do not constitute significant bandwidth traffic on your site. Look at your active server logs to see if you can find any kind of pattern to the entries in the error log...which might provide info regarding scrapers, mal-links, etc. Better place to spend your time. The error.log is links already dealt with.

[helenp]

I do because I have a strange 500 error code wich raises my memory to max, and then I saw this, that ip belongs to one of swedish biggest internet company. And all 4 of the ips I checked are blacklisted in spamhause.

[tangor]

The 5x and 4x status returns no site content, correct? If true, I'd move along to all the other problems that websites experience from time to time... in the ACTIVE SERVER.LOG :)

[lucy24]

Why are you spending time in your ERROR log? Those entries have already been handled by the server and do not constitute significant bandwidth traffic on your site. Look at your active server logs to see if you can find any kind of pattern to the entries in the error log...which might provide info regarding scrapers, mal-links, etc. Better place to spend your time. The error.log is links already dealt with.

This isn't always true. A 403 in your access log generally means door slammed in their face, "client denied by server configuration", end of story. But it can also mean that you goofed in your .htaccess and the 403 is the end result of an infinite rewrite with the server saying Enough Is Enough after (generally) ten iterations. If this is the case you do need to take the occasional look in the error log, find and isolate the problem.

Is anyone having trouble figuring out how I know this?

[tangor]

Not me. :)

.htaccess is sparely used (about 200 lines). Games played are seldom and rarely subject to error. (grins again)

On my side a 403 is specific denied and intended, which is different than a 404 (not found).

[SteveWh]

When a request says that the referer is your index.php, but you don't have an index.php, it just means that they are faking the referer (lying about it). Therefore, it is a bogus request to start with, and you need not worry about it.

It's not related to (and not caused by) your .htaccess line that tells the server to treat html as php pages.

[helenp]

I downloaded the logfile and searched for the ip and there were nothing to see, only a 404 error (the 406 did not exists) and the owner of the host said this last night:
"We don't recommend allowing blacklisted ip's as thats usually where spam and hack attempts come from."I went and looked through logs and found the site to be rejecting them based of the RBL listings and not the browser related so they'll still be blocked".

I checked the last 406 I could see in error log as the errors quickly disappears and all where in spamhaus xbl and when I read spamhaus xbl faqs they tell hosts not to block these ips to access the server as they will be blocking inocent people. So they were blocking internet providers that were on spamhaus list saying spammers and hackers came from there.
Angry the host last night he said he has disabled blacklisting to access my account.

Regarding the 500 error its related to my chatprogram that is offered by cpanel-fantastico that I been using since may and had problem when they installed mod_something last summer as they had problems with wordpress, at that moment I got 500 error every time I opened the chat and had to argue with the host that it was not the program that was bad, suddenly the bug was fixed however the host said they had not changed anything.

On the 6th january the host migrated the server to a new hardware. On the 13th january I reported that I could not access my password protected directory as I got a 500 error and the emails didnt work well. They had cpanel to fix it and was fixed some hours later.
Not much later I reported that sometimes when I opened chat I get 500 error and had to refresh, however that became worse and worse and I started to look and saw in errorlogfile that the chat gave error 500, in resourge usage I had sometimes max memory usage in 384 m while the average was maybe 1.x something. This has been worse and worse, now I cant open the chat as I get 500 and memory rize to 384.0 mMEM, I have taken away the links to the chat to see if I get high numbers without it, and I do get but not that often.
For exempel this is from about 4.00 in spain where I live and normally we have few visitors as we are in europe:
01-20 20:00 01-20 21:00 0 20 0 1 20 624K 255.6M 384.0M
As you can see the average is 624k but the max reached is 255.6.
Number that sounds just crazy to me, hour when I was sleeping and I dont have any automatic processes, and of course my chatpage was closed as I was sleeping and pc shut down.
And the host says that its a script error, but I have not done any change at all to the chat, its not my script I dont even understands it, and I seen in resourge usage having a look for last 30 days that it started to show high max on the 13th between 16-17.00 that is about 24.00-1.00 in spain. I had ever had so high numbers before.
Any help is appreciated.

[helenp]

trying to track the bug, the chat is desinstalled and no links on page, however between 6:15 and 6:20 (servertime) I got average 3.5 and max 207 wich I think is a high max number especially as saturday is a very bad day.
I checked the error log same time and strange, host disabled blacklisted 406 error, however I keep getting some, but the one I seen does not have any referrer, and in that timeperiod there is one, dont know if a coincidence or not, also one ip has get //directory and got a 200, // is not correct is it?
Anyway as it is not that much I will past all between those 5 minutes:

207.46.13.114 - - [21/Jan/2012:06:09:39 -0700] "GET /robots.txt HTTP/1.1" 200 277 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
80.174.72.31 - - [21/Jan/2012:06:10:17 -0700] "GET /live/admin/index.php HTTP/1.1" 200 1215 "-" "Mozilla/5.0 (Windows NT 5.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1"
65.52.109.146 - - [21/Jan/2012:06:15:11 -0700] "GET /svenska/casa_blanca5_privat_villa_marbella.htm HTTP/1.1" 404 1834 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "GET /espanol/formulario_reserva.php?propiedad=Alcazaba_1 HTTP/1.1" 200 9370 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "GET /images/arrow_right.gif HTTP/1.1" 200 73 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "GET /espanol/css/availabilitynuevo2.css HTTP/1.1" 200 1470 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "GET /espanol/css/nuevo7.css HTTP/1.1" 200 5918 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "GET /espanol/css/validate.css HTTP/1.1" 200 477 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "GET /espanol/includes/validate_all.js HTTP/1.1" 200 66049 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:18 -0700] "POST /espanol/formulario_reserva.php HTTP/1.1" 302 9360 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:20 -0700] "GET /espanol/paginassub/formulariohecho.htm HTTP/1.1" 200 1929 "http://www.exampel.com/espanol/formulario_reserva.php?propiedad=Alcazaba_1" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:20 -0700] "GET /espanol/formulario_reserva.php?propiedad=Alcazaba_1 HTTP/1.1" 200 9370 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /espanol/ HTTP/1.1" 200 20758 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/testsol9.png HTTP/1.1" 200 1523 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/bandera_sueca_.gif HTTP/1.1" 200 1141 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/bandera_inglesa_.gif HTTP/1.1" 200 1260 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /espanol/ticker.js HTTP/1.1" 200 1802 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /espanol/new_arrow_down.gif HTTP/1.1" 200 270 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /espanol/new_arrow.gif HTTP/1.1" 200 277 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1costa_nagueles2.jpg HTTP/1.1" 200 4847 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1cortijo_blanco_1.jpg HTTP/1.1" 200 4588 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1dama_de_noche.jpg HTTP/1.1" 200 4819 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1el_higueral3.jpg HTTP/1.1" 200 4667 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/top_icon.gif HTTP/1.1" 200 49 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/visa.gif HTTP/1.1" 200 431 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/mastercard.gif HTTP/1.1" 200 528 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/cardeurocard.gif HTTP/1.1" 200 928 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /images/4b.gif HTTP/1.1" 200 680 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1villa_jorgepool.jpg HTTP/1.1" 200 4250 "http://www.exampel.com/espanol/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /espanol/index.htm HTTP/1.1" 200 20795 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1marbelcentre_1.jpg HTTP/1.1" 200 4987 "http://www.exampel.com/espanol/index.htm" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1hacienda_nagueles.jpg HTTP/1.1" 200 4891 "http://www.exampel.com/espanol/index.htm" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1solmarbella6.jpg HTTP/1.1" 200 4579 "http://www.exampel.com/espanol/index.htm" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
65.213.70.116 - - [21/Jan/2012:06:17:21 -0700] "GET /thumbnails/1la_zambomba.jpg HTTP/1.1" 200 4518 "http://www.exampel.com/espanol/index.htm" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
121.160.74.206 - - [21/Jan/2012:06:19:39 -0700] "GET /espanol/_disponibles_marzo.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:40 -0700] "GET //espanol/_disponibles_marzo.php HTTP/1.0" 200 55525 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:41 -0700] "POST //espanol/_disponibles_marzo.php HTTP/1.0" 302 147 "http://www.exampel.com/espanol/_disponibles_marzo.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:41 -0700] "GET //espanol/paginassub/formulariohecho_casas.htm HTTP/1.0" 200 2094 "http://www.exampel.com/espanol/_disponibles_marzo.php" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:41 -0700] "GET //espanol/paginassub/formulariohecho_casas.htm HTTP/1.0" 200 2094 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:41 -0700] "GET //espanol/_disponibles_marzo.php HTTP/1.0" 200 55525 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:42 -0700] "GET //espanol/ HTTP/1.0" 200 20757 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:43 -0700] "GET //espanol/index.htm HTTP/1.0" 200 20781 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:43 -0700] "GET //svenska/index.htm HTTP/1.0" 200 20658 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:44 -0700] "GET //index.htm HTTP/1.0" 200 20424 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:44 -0700] "GET //espanol/libro_de_visitas_2011.htm HTTP/1.0" 200 21028 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:44 -0700] "GET //espanol/nosotros.htm HTTP/1.0" 200 19025 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:45 -0700] "GET //espanol/pacos_pros.htm HTTP/1.0" 200 18340 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:45 -0700] "GET //espanol/banana_beach_apartamento_primera_linea_playa.htm HTTP/1.0" 200 22072 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:45 -0700] "GET //espanol/costa_nagueles2_apartamento_marbella.htm HTTP/1.0" 200 32767 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:46 -0700] "GET //espanol/hacienda_nagueles_apartamentos_milla_oro.htm HTTP/1.0" 200 19441 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:47 -0700] "GET //espanol/la_zambomba_apartamento_playa_marbella_centro.htm HTTP/1.0" 200 31028 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:47 -0700] "GET //espanol/marbelcentre1_apartamento_marbella_centro.htm HTTP/1.0" 200 33968 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
186.125.202.59 - - [21/Jan/2012:06:19:49 -0700] "GET /bookingform.php?propiedad=Sol_Marbella_6 HTTP/1.0" 200 9453 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
50.16.247.234 - - [21/Jan/2012:06:19:49 -0700] "GET //espanol/marbella_del_mar_apartamentos_centro.htm HTTP/1.0" 200 19696 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:49 -0700] "GET //espanol/sol_marbella6_apartamento_marbella_centro.htm HTTP/1.0" 200 32901 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:50 -0700] "GET //espanol/solymar_2_apartamento_marbella_centro.htm HTTP/1.0" 200 32921 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:51 -0700] "GET //espanol/las_canas_beach_apartamentos_primera_linea_playa.htm HTTP/1.0" 200 19683 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:52 -0700] "GET //espanol/casa_blanca4_villa_privada_marbella.htm HTTP/1.0" 200 34837 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:53 -0700] "GET //espanol/el_higueral_3_villa_marbella.htm HTTP/1.0" 200 33524 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:54 -0700] "GET //espanol/el_higueral_4_villa_marbella.htm HTTP/1.0" 200 34512 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
50.16.247.234 - - [21/Jan/2012:06:19:54 -0700] "GET //espanol/xaviera_villa_privada_marbella.htm HTTP/1.0" 200 37317 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"
186.125.202.59 - - [21/Jan/2012:06:19:55 -0700] "POST /bookingform.php HTTP/1.0" 302 9439 "http://www.exampel.com/bookingform.php?propiedad=Sol_Marbella_6" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
186.125.202.59 - - [21/Jan/2012:06:20:00 -0700] "GET /pages/formdone_casas.htm HTTP/1.0" 200 886 "http://www.exmpel.com/bookingform.php?propiedad=Sol_Marbella_6" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
186.125.202.59 - - [21/Jan/2012:06:20:06 -0700] "GET /bookingform.php?propiedad=Sol_Marbella_6 HTTP/1.0" 200 9453 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
186.125.202.59 - - [21/Jan/2012:06:20:09 -0700] "GET / HTTP/1.0" 200 20565 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10"
186.125.202.59 - - [21/Jan/2012:06:20:14 -0700] "GET /index.htm HTTP/1.0" 200 20486 "-" "Mozilla/5.0 (Windows

[lucy24]

one ip has get //directory and got a 200, // is not correct is it?

I think there's a command that gets rid of superfluous leading slashes, because I get this sometimes too. If it's the only error in the request, it's handled as usual. But humans don't vroom, vroom through your site that fast; these double-slashes are all from 50.16.247.234. A lot of people have the 50.16.0.0/14 range blocked; it's robot territory. (AmazonAWS, I think.)

I was going to say that I don't see any errors in this slab (quick-and-dirty search for " \d+ \d+ " with leading and trailing space) although I note that all POST request get a 302. I hope that's intentional. But then this caught my eye:

121.160.74.206 - - [21/Jan/2012:06:19:39 -0700] "GET /espanol/_disponibles_marzo.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"

How do you manage to get errors with no follow-up filesize? Is that your host being weird?

[helenp]

121.160.74.206 - - [21/Jan/2012:06:19:39 -0700] "GET /espanol/_disponibles_marzo.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5"

How do you manage to get errors with no follow-up filesize? Is that your host being weird?

Didntīt noctice you correct, didnīt noticie there are no filesize...long since I seen a log.
However the pages are just fine.
I done all my scripts and nothing happened to mMME and btw I have not changed anything for a long time.
And as far as I understand the size of the document does not have anything to do with memory only bandwith or am I incorrect?

That entry you pasted is the one that the ip is in spamhaus and on that entry I have a 406 error.
He blocks blacklisted ips with a 406 error, however I dont think its correct as those ip ara internet providers, and I read spamhause instructions and it said you should not block the ips to enter the server, I suppose to send emails but no to block to view webpages.
He disabled it on my site and yes I do get a lot of spam in my contact form, but that is preferable to loose clients. But its not well done as I still get some 406 error, and when I check spamhouse they are listed in the last one, the one that the serve have virus.

[helenp]

just got another high max memory number 242m during 5 minutes.
And I have a ton of 404s for pages I deleted some months ago for the same 5 minutes. The ip belongs to SINGLEHOP.
So no doubt the bots rize the max. memory, that means 2 aggresive bots at the same time and I reach max memory wich will give a 500 error to visitors.....and this having the chat desinstalled....if I use it I get max memory all the time. I have desinstalled and installed a new version of the chat (using cpanel-fantastico) and its just the same, 500 errors and reachin max memory.

So obviously or the bots being aggresive rize the max memory or the errors rize it.
Doesnīt bots consume bandwith only?
I thought memory is used if using applications, mysql queries etc....

[lucy24]

Any php page (or jsp, asp etc.) will use a little bit of memory simply by building the page. The bandwidth kicks in at the next stage, when it sends the page-- along with any files such as stylesheets or images that the page asks for. Normally only humans get this extra stuff; robots concentrate on pages. More memory, less bandwidth compared to humans.

If a robot puts in a request for something with a query, then the php (or whatever) will use up a certain amount of memory trying to make sense of the query even if the php function ends up not being able to do anything with it. Similarly if the page requires a query and there isn't one, that will also take up memory. If you have a lot of this happening-- either fake queries or absent queries-- you need to intercept them in htaccess and block the request before it gets to the page. If a 403 is coming from the php page itself, that's memory being wasted.

A 406 error is very weird and probably wrong. It is definitely not supposed to mean "I don't like your face". That's what a 503 is for. A 406 is more like "I'm sorry, we don't carry that in your size".

And I have a ton of 404s for pages I deleted some months ago

Slightly off topic, but if a page used to exist and you deleted it on purpose, you shouldn't let it return a 404. Either redirect to a good replacement page, or use a 410 ("Gone"). A 410 will also get rid of robots faster than a 404.

long since I seen a log

Look at them periodically. You can get more information out of raw logs than out of fancy analytics. But you have to look in the right places. I've got a log-wrangling routine that does a lot of things semi-automatically. That includes getting rid of all image/css/js requests that belong to a page; and ignoring all authorized robots and all 403s. What's left tells you the two things you really need to know: what your human visitors are doing, and what kinds of unauthorized robots are snuffling around.

[helenp]

Thanks,
Yes I thought also afterwards that as is php it can use memory, however hasnt had that high numbers between the 6th (when migrations was done) and 13th as I got from the 13th until now and I am imagine there were bots between the 6th and 13th also.
On the night of the 13th I started to get error 500 on my chat, and sometimes high levels of max memory usage and now I cant even open it even if I just installed a fresh version from cpanel-fantastico. I did the test to install the same program from cpanel on another account that is not being used and it got high level as soon as I opened the chat and it tried to connect to see if anybody where on the site. To me something is wrong,and sounds very very strange. I had ever worried as I ever reached any max levels before.

And I wonder is it normal practice to block ips to visit your website just because they are in a blacklist? Blocking for exampel swedens biggest adsl provider using mod_security giving a 406 instead of an 403?

I know they normally blocks those ips from receiving emails but ever heard of blocking for visitng websites...For exampel I dont use the email my adsl provides me with, so I am loosing clients while they do that. I checked all 406 I got with spamhaus and they are all in the lost, most in the last one that is for having virus, and in there guidelines they tell not to block inocents.

Also I cant do antoher error instead of 404 as I often drop pages and do others, the list would be very long with the time as its properties and we take new ones and drop others.

[helenp]

Just checked my error log, had 3 406, all 3 in spamhaus,
the first dont know what is, is chinese, the second is I think an important internet provider, not sure:
Comcast Cable Communications Holdings, Inc
the third is also from comcast cable.

This is spamhaus guidelines for one of blacklisting:
Should I use the PBL to block access to my webserver?
No! A listing in the PBL does not mean there is anything 'wrong' with the IP address or end user. A PBL listing does not mean an address is an open proxy or run by a spammer. All it means is that the IP address has been designated as 'not allowed to make direct-to-MX SMTP connections'. The majority of legitimate connections to webservers come from IPs listed in PBL. Please do not block innocent users.

And this is for the most frecuent one:
Should I use the XBL to block access to my webserver since it means that the IP address has a virus or open proxy?
A listing in the XBL does not mean this. It means that at one time the IP address may have had a virus or open proxy.

The XBL contains mostly dynamic IP addresses, meaning the user you would be blocking is probably not going to be the user with the exploited computer. Please do not block innocent users.

If you still feel you must use the XBL in this way, do not refer users back to Spamhaus. You must deal with blocked users yourself. Either by giving them a point of contact, or perhaps by instituting a CAPTCHA + cookie system to screen out spam-bots.

Maybe this is normal practice in shared hosting, no idea, but I dont think its the correct.

He just closes the door using a 406

[lucy24]

It's definitely not what my host does, and they've got something like four trillion hosted domains. (I forget the real number, but it has more zeros than you would think possible.) Very very rarely a visitor gets hit with a 503 if it is an IP belonging to a known evil robot. They've also got something in their config file that looks for certain patterns.

But in general it's up to the individual user who they want to block. Lots of robots are annoying and troublesome but hardly any of them are actively malign. And most of the really nasty ones keep skipping to a new IP so you can't block them in advance. As far as I know, I've never got a 406 error. Not even with robots who come looking for nonexistent php pages with suspicious queries. I had to go look up 406 just to find out what it is.

When I first got this host, I was told that I wouldn't be able to send e-mail from my domain. That is, I can send it just fine. But the recipient may not get it, because a lot of ISPs block e-mail coming from anything that isn't another ISP. That's probably fairly common. But that's e-mail. Wholesale blocking of every IP under the sun seems extreme. Even the Ukraine sends human visitors sometimes. Are you sure you have the best possible host? :)

Oh, and yes, Comcast is a major ISP. They may do servers on the side, but I think their main business is with humans. I definitely would never think of blocking them wherever I see them.

[helenp]

It's definitely not what my host does, and they've got something like four trillion hosted domains.Very very rarely a visitor gets hit with a 503 if it is an IP belonging to a known evil robot. They've also got something in their config file that looks for certain patterns.

Are you sure you have the best possible host? :)

I suppose you mean you get a 403 not a 503.

Well i have been pleased with the host until now as before I had many problems with greylisting emails coming back and when he bought they host and migrated to new server the greylisting problem disappeared. However every time there were a bug that was not easily demostrated there fault, I have to argue and he keeps telling me to hire a programmer....bad manners, but finally the things got fixed.

Anyway he does not seem sure about the 406 as last night he said diferent things, and all of them I checked, only 1 day looks like bots, but the true is there are more bots than human.
I have added a crawl-delay: 10 to see if becomes better as the one I posted before says it do respect the crawl-delay, but as I said, I didnt have any number that high before.
And what is bothering me also is the chat that do take resources, but not that much.
So I suppose if not changed in a coupple of days I better look for a new host.
I know you cant recomend here, but I need a good and cheap as we are not a heavy traffic site.
Thanks for your help.

[helenp]

Think I better look for a new host as he needs help, however it scares me a bit, and dont know if stay in usa or migrate to europe, posted in webmaster general about it:

Just wanted to share his stupidez:
About blocking with 406
1- "We don't recommend allowing blacklisted ip's as thats usually where spam and hack attempts come from.
Im closing this issue as we will no longer support your site except for outages due to your asking to have security removed."
I did not ask for having it removed, I was doubting if was correct....

About the 500 error and max memory when opening chat:
2-"I've just installed it on a account that does not have cloudlinux and it immediately ate up over 512mb ram out of cloudlinux. Its not a bug, its a resource intensive script."
Wow,a resource intensive script since 13th january that had not changed on my part, and is offered by cpanel.

Also about the chat, showing he lost his head.
3-I can't answer why it worked and why it doesn't now. Nothing changed between the 6th and the 13th.
The issue you had on the 13th was not necessarily a bug but wrong set of commands issued for fixing permissions.
I'd contact the developer of the script and advise them it was working on the 6th on php 5.3 and cloudlinux with 384mb allocated memory and on the 13th it stopped working with no changes and see what input they might have.
Lol, first he said nothing changed then he says they fixed permissions...
And the best, to ask the developer why his script stopped working on a server......knowing the developer has not changed anything. (on the 6th there where a migration and on the 13th I had 500 error on my password protected directory)

Lol, lol and lol

[lucy24]

I suppose you mean you get a 403 not a 503.

No, I really meant 503, because I was talking about the exceptional case where the host does step in and slam the door on a visitor. That only happens once or twice a year. The ones I lock out via my own htaccess get 403.

I guess lol is about all you can do when the alternative is to beat your head against the wall and scream in rage :)

I'd forgotten about it, but there is an ongoing thread about the Dark Side of Comcast over in the Search Engine Spiders forum.

[helenp]

Yes could be, but its not the correct way, I seen so many diferent ips with 406, suppose its an automatic connection with spamhaus, ip on the list, ip that is blocked. Swedens biggest mobile and normal adsl provider got a 406 also, and yes it was on the blacklist, that one that said there were people that were sending virus....
I did a search for my hosts ip and I found it on 3 lists lol, not in spamhaus but in others. And I reported emails I got back due to blacklisting more times than I have fingers on my hand.

[helenp]

Lucy,
I wonder what the host have done, have not received any spam email filling in my form, no 406 in error log at all anymore.
Did a search in logfile for a 406, 403 and 503 and there is none. I suppose they should appear there.
Could you please send me in a privat message the name of your host so I can check if it can be of any interest to me? Thanks in advance.
Helen

[helenp]

HI, Even though I am definitively looking for a new host I like to know, that way I learn and things wont repeat.

In October I did a big change in website where I added a slideshow with bigger images. Didnt have any problems, these I pregzipped not to use resourges. And I remember when I where checking header that connection said keep-alive.
I just checked my header and it says connection: closed.
As I have many images and files on some pages that means for every image or file a new request is sent and apache opened?
However I read that having connection closed higher the cpu usage and not the memory usage wich is my problem at this moment.
For examepel between 24.00 and 1.00 when we have few visitors as its night in europe last night I reached max. mmem, thing I ever did before.
Can the connection closed be the reason.
I checked my raw log file and cant see anything change, only a few robots have not many files, but what at least looked as visitors looked at villas pages wich have many photos as they are big = many requests.

So if somebody can explain as cant find what takes a lot of cpu and memory on a dynamic site. And if images takes memory and cpu also.
Thanks in advance.