Blocking visitors

[qimqim]

I'm being bombarded with visits from an outfit in Russia that shows in Google Analytics as econom.co / referral. So, I added them to the .htaccess file, but theu keep coming.

Maybe I did something wrong in the file

#1 # block visitors referred from indicated domains

RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} make\-money\-online\.7makemoneyonline\.com [NC,OR]
RewriteCond %{HTTP_REFERER} darodar\.com [NC,OR]
RewriteCond %{HTTP_REFERER} econom\.co [NC,OR]

RewriteCond %{HTTP_USER_AGENT} libwww-perl
RewriteRule .* – [F]

Could you have a look, please?

I've seen them listed as Econom.co. Could the capital "E" make a difference?

Thank you

[not2easy]

The semalt referrer spambot does not stop visiting just because you deny it access, it will show up in your logs until they change their habits, you cannot stop them from trying. In your access log you can see why they don't show in GA: because they did not get any page, they got a 500 error. That is what is says here:

"GET / HTTP/1.1"

500 That should be a 403, but we don't know how you have set up your 403s, somewhere it is not quite right. Although a 500 error prevents access, they may try longer than if they got a 403.


There is a basic difference between user agents and "referrers" The Mail.RU UA is a real bot, so yes, it will look at robots.txt. Being a real bot, it has a home, so you can block that with either UA or IP. I use the IP in case they send their cousin in wearing a different hat. To block them and all their cousins:

217.69.128.0 - 217.69.135.255
MAILRU-NET
deny from 217.69.128.0/20

You can add that where you deny from env=spambot

[qimqim]

Hi not2easy

The semalt block is in the initial post and above,at the top of this page also.

Been doing some homework (finally...)

First, it turns out that all this ...ilovevitaly business is just a fake that hits GA without ever accessing my site. Read

http://www.blackmoreops.com/2014/12/19/darodar-com-referrer-spam/#tc-comment-title

Now what is hitting me and I have not been able to get rid of is this outfit which I put in the htaccess but is not working. Thre plenty of instances of them in the log. What is wrong with the code in the htaccess?

179.110.3.28 - - [12/Dec/2014:13:46:34 -0700] "GET / HTTP/1.1" 200 10974 "http://make-money-online.7makemoneyonline.com/money.php?u=http://pintotours.net" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"

[not2easy]

Something in your htaccess is not working the way it should or you would not have

"HTTP/1.1" 500"

in your logs. It might be helpful to clear up the errors that are preventing your efforts from serving a 403 if you do post your htaccess file, with the domain name replaced with example.com. You can post lines from your access logs also - again with your domain replaced using "example.com". Just don't dump the whole thing as it is messy to clean up. Best practice is to use [ code ] tags so it doesn't mess up the format.

It may be a tiny little detail that is not quite right that would keep serving a 500 error where it should be a 403.

[wilderness]

In addition to what not2easy has suggested?

I'd suggest that you review more current examples of these visits from your raw logs.
1) This thread started on Dec 18th and just today your providing visitor examples from your logs of Dec 7 & 12, rather than Dec 26th (or at least after the 20th.

[lucy24]

217.69.133.251 - - [07/Dec/2014:00:12:05 -0700] "GET /robots.txt HTTP/1.1" 200 365 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.248 - - [07/Dec/2014:00:12:07 -0700] "GET /Pinto/insurance.html HTTP/1.1" 200 3311 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)"
217.69.133.21 - - [07/Dec/2014:00:12:10 -0700] "GET /Asia/Indonesia/RamadaBintangMap.html HTTP/1.1" 200 1614 "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)" 

Mail.RU is a legitimate search engine and that is their actual IP. Admittedly the URL given in the UA string is not much help, as it redirects (slowly!) to a Russian-language page in which I can spell out "robot" in Cyrillic. Make your own decision about whether you want to let them in; I currently let them have pages but not images. They do appear to obey robots.txt

qimqim, ask your host about those 500's. Normally a 500-class response is server-side; if you're not doing it yourself, they may be issuing their own lockouts, for example via mod_security. It's also possible that something in your htaccess is wrong, leading to 500 errors on requests that match a particular pattern. But it's pretty hard to create an htaccess that only gives a 500 error on specified requests; most errors will bring down the whole site. Either you're doing it by accident or your host is doing it on purpose. Find out.

First, it turns out that all this ...ilovevitaly business is just a fake that hits GA without ever accessing my site.

Oh, sneaky. Since their sole purpose is referer spam, there's no point in physically requesting anything on your site when they can achieve the same purpose by simply hitting your Analytics code. They probably made one preliminary visit to your site to get the Analytics link. If you had your raw logs dating back to the dawn of time, you'd be able to check.

You should check your Analytics settings and see how to block them outright or, at least, conceal their visits from your view. Out of sight, out of mind. This is happening on Google's server, so you can't simply ban them in your own htaccess. If nobody reading this thread is chummy with GA, wander next door to the Analytics, Tracking and Logging [webmasterworld.com] subforum. Someone will know. (I don't personally know because I use a different analytics package.)

[qimqim]

Hi Wilderness

Replying one at a time

The last entry in the log for the 7makemoneyonline is adted the 20th but they come nearly evryday up until then

186.225.34.29 - - [20/Dec/2014:14:16:47 -0700] "GET / HTTP/1.1" 500 561 "http://make-money-online.7makemoneyonline.com/money.php?u=http://pintotours.net" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"

[qimqim]

Hi not2easy and Lucy

Thanks to both-

Here goes the current htaccess
semalt,buttons and makemoneyonline are returning all 500

#Use PHP5.4 Single php.ini as default

AddHandler application/x-httpd-php54s .php

#
AddType text/x-component .htc


#Do not allow access to the directories -For security reasons, Option followsymlinks cannot be overridden.

Options -Indexes +SymLinksIfOwnerMatch
RewriteEngine on




#1 # block visitors referred from indicated domains

RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} make\-money\-online\.7makemoneyonline\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} libwww-perl
RewriteRule .* – [F]

#2 bandwidth theft
RewriteCond %{HTTP_REFERER} !^http://example\.net/

RewriteRule .*\.(jpe?g|gif|png|bmp)$ - [F]

#3 redirects from file that changed name

#3a
RewriteRule ^Pinto/oldindex\.html http://example.net/Pinto/oldindex.php [R=301,L]

# 3b
RewriteRule ^Asia/Indonesia/bali\.html http://example.net/Asia/Indonesia/bali.php [R=301,L]

# 3c
RewriteRule ^Asia/Indonesia/indonesia\.html http://example.net/Asia/Indonesia/indonesia.php [R=301,L]

# 3d
RewriteRule ^Americas/DomRepublic/DomRepublic\.html http://example.net/Americas/DomRepublic/StoDomingo.php [R=301,L]



#4a index redirect

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.html
RewriteRule ^(.*)index\.html http://example.net/$1 [R=301,L]


#4b domain-name canonicalization redirect

RewriteCond %{HTTP_HOST} !^(example\.net)?$ [NC]
RewriteRule ^(.*)$ http://example.net/$1 [R=301]

#5
# BEGIN EXPIRES
<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access 1 week"
ExpiresByType text/css "access plus 1 week"
ExpiresByType css/js "access plus 1 week"
ExpiresByType text/plain "access plus 1 week"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType text/javascript "access plus 2 month"
ExpiresByType application/javascript "access plus 1 week"
ExpiresByType application/x-icon "access plus 1 year"
</IfModule>
# END EXPIRES



<IfModule mod_deflate.c>
<FilesMatch "\.(js|css|html|php)$">
SetOutputFilter DEFLATE
</FilesMatch>
</IfModule>


#Set charset

<filesMatch "\.(htm|html|css|js|php)$">
AddDefaultCharset UTF-8
</filesMatch>

[edited by: qimqim at 9:54 pm (utc) on Dec 27, 2014]

[qimqim]

Hi wilderness

I beg your pardon...

You were correct in asking for a later example. Possibly I only added the block for moneyonline later and it is now showing 500

[qimqim]

Hi all

I've been looking at the files in my root directory and there are some files that I inherited from the Host when I joined. They are 400, 401,403,404, and 500 all .shtml and the 500 in .php also

Could this be related to the fact that these blocks are returning 500 instead of 403?

[wilderness]

Here goes the current htaccess
semalt,buttons and makemoneyonline are returning all 500

Your htaccess does NOT include any
ErrorDocument 403 /yourFileName.html
ErrorDocument 404 /yourFileName.html
ErrorDocument 410 /yourFileName.html

Thus they are being denied, however most likely going into a loop for the non-existent files.

've been looking at the files in my root directory and there are some files that I inherited from the Host when I joined. They are 400, 401,403,404, and 500 all .shtml and the 500 in .php also

Could this be related to the fact that these blocks are returning 500 instead of 403?

DELETE all these at once.
It's seems to be a current practice by shared hosts and the file are used in duality to provide the custom errors, however the 2nd use is to promote their services on your dime. (your paying them for hosting and there's no contractual agreement to promote their services in this manner).

Generally hosts that uses these custom error methods also require an adjustment in CP to designate your NEW custom-error-pages.

[qimqim]

Hi wilderness

thanks, but whre exactly to I put these 3 lines in the htaccess file?

ErrorDocument 403 /yourFileName.html
ErrorDocument 404 /yourFileName.html
ErrorDocument 410 /yourFileName.html

I did not get totally the last bit of your post.

also require an adjustment in CP to designate your NEW custom-error-pages.

I will delete the redirect files from the root directory, but what else do I need to do?

[lucy24]

Your htaccess does NOT include any
ErrorDocument 403 /yourFileName.html
ErrorDocument 404 /yourFileName.html
ErrorDocument 410 /yourFileName.html

Thus they are being denied, however most likely going into a loop for the non-existent files.

Oh, ###, this is going to be one of those days when I disagree with everyone. The lack of an ErrorDocument directive won't automatically lead to 500-class errors. In fact, on shared hosting there is probably a section of the config file that says something like

ErrorDocument 403 /forbidden.html
ErrorDocument 404 /missing.html

et cetera, and then there will also be a <Files> envelope for "forbidden.html" that says

Allow from all

, precisely so the server can send out the 403 document even if the request would otherwise be blocked. These lines, assuming they exist, are in the server config file. You can't change them, but you can override them if you know what you are doing.

Now, if the server expects to find a document called "forbidden.html" in your root, and doesn't find it, then it will go looking for a document called "missing.html" (note that I'm just hypothesizing these names) in the same place and, upon not finding that, it will go into an infinite loop ending in a 500 error. This is not entirely bad, since it does mean the unwanted visitor never gets what they asked for; it just means the server has to do a lot of extra work-- 30 internal requests instead of 2.

So what's missing isn't necessarily the ErrorDocument directive. It's the Error Document itself. If the documents already exist, you are free to replace them with custom documents of your own, using your own stylesheets and so on, so long as they have the same name. (If they weren't supposed to be edited/replaced, they wouldn't be in your own directories where you can get to them.) You only need an ErrorDocument directive of your own-- and possibly your own Files(Match) envelope-- if you're using something other than the default error document (either a different URLpath, or a different name).

Can you get hold of your error logs? They should live in the same place as your access logs. Unlike access logs, error logs should also show errors arising from internal requests. So a 500-class error should be pretty easy to spot.

[qimqim]

Hi Lucy

Cannot find any 500 in error log which is shared.
I have uploaded the latest file here

[wikisend.com...] cPanel - example.net.htm

[edited by: phranque at 9:05 pm (utc) on Dec 28, 2014]
[edit reason] no personal domains please [/edit]

[lucy24]

Error logs won't give the numerical code of an error. But they'll say in words what happened. For example "client denied by server configuration" means a 403. (This is the single least useful line in error logs. But weed it out and what's left over, if anything, is informative.) This line seems to be entirely absent from your quoted logs. Don't know if that's coincidence (the quoted part only spans about one minute) or a problem. Depending on logging level (out of your control), the server can choose not to list 403s.

Is there no way for you to download the actual log files to your hard drive instead of viewing them through cPanel?

Your error logs do contain some useful information, though, notably lines such as

RewriteCond: NoCase option for non-regex pattern '-d' is not supported and will be ignored.

Didn't I say something about [NC] earlier? Here fortunately it only means that the flag will be ignored, not the entire line-- or entire RewriteRule, or entire htaccess. But you're racking up an awful lot of this category of error, since the lines have to be read on every request.

There is also recurring refererence to mod_rbl and one ModSecurity. This may explain the 500 errors: If your host uses mod_rbl and/or mod_security to deny requests, it's their choice what error code to return. They might well choose "500".

There are a lot of php errors, some of which may well fall into the "Just sayin'" category. (My MAMP logs have a lot of "Error: I can't find such-and-such cookie" because the code says to look for the cookie and take different actions depending on whether you find it.) If you suspect anything is going wrong on the php side, ask next door in the php forum. penders or someone like him will answer articulately.

:: pause to paste server errors into text editor and sort by category ::

It looks as if the server is returning a 501 for at least some mod_security requests: look at the lines involving 200.109.230.177.

I do find one batch of infinite-loop errors, though the error log is surprisingly unhelpful about the cause. The relevant IP is 180.76.5.abc, which is Baidu-China.

I also see what looks like someone at 187.17.106.abc trying to use you as an email proxy. I don't know anything about the IP except that it's in Brazil.

If you choose to paste-in any part of your error logs to WebmasterWorld forums, replace "servername.hostname.com" with, ahem, "servername.hostname.com" in the same way that you'd replace your own domain name with "example.com".

:: final detour to compare RBL IPs with my personal lockouts ::

Oh, ###, how did I fail to lock out that OVH range even though I've got it clearly identified as OVH*? Looks like I've got all the others blocked.


* I don't know French so I don't know how OVH gets to be an abbreviation for "Francophone robot". I just know that some of them live in Montreal.

[onlineleben]

Just two little observations about the darodar-spam:
1) the referer listed in GA contains the GA-code in the subdoman-name
2) just today a new one popped up: forum.topicXXXXX.adviceforum.info (XXXX are numbers that I cannot associate with any of my google accounts (GA, AS) yet.

To me it looks like a little more than referer spam.
Any ideas what they have in mind and also how long it will take google to react on things like this?

[Leosghost]

Don't need to speak French..
OVH = Often Very Harmful..
Which is why we blocks 'em, wherever they claims to be from..n'est-ce pas ;)

[lucy24]

Which is why we blocks 'em, wherever they claims to be from

In general I take a wait-and-see attitude toward server farms. After all, everyone starts out small, and some robots are benign. (I'm currently entertaining a Finnish one whose behavior to date has been impeccable.) But a select number of hosts, such as OVH and Hetzner, are Shoot On Sight.

the referer listed in GA contains the GA-code in the subdoman-name

Hm now that's interesting. Does that mean each one started out by visiting your domain to learn the code? Or are GA codes sequentially assigned (like email addresses* in those halcyon early years) so a robot just has to ask for site1234, site1235, site1236 and so on through the list? I agree it does seem as if G### should eventually figure it out and execute some kind of clampdown. Don't some people actually use GA data as an objective external source of information about a domain's current traffic and hence profitability?


* I was the 24th person with my exact initials at X University.

[qimqim]

Hi Lucy

RewriteCond: NoCase option for non-regex pattern '-d' is not supported and will be ignored.
Didn't I say something about [NC] earlier? Here fortunately it only means that the flag will be ignored, not the entire line-- or entire RewriteRule, or entire htaccess. But you're racking up an awful lot of this category of error, since the lines have to be read on every request.

Do you mean by the above that I should delete the 3 [NC] in the file?

#1 # block visitors referred from indicated domains

RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} make\-money\-online\.7makemoneyonline\.com [NC,OR]

RewriteCond %{HTTP_USER_AGENT} libwww-perl
RewriteRule .* – [F]

Regarding the below, please note that the log says at the top that it is
shared by all sites on that Host ip.

The MAIN error_log is a shared log meaning that errors from all websites hosted on the server will be displayed as well as those for your website, be sure to reference your IP when searching this log

So, why do you think it is connected with mine? According to the IP at the top of the page, none of the entried relate to me.

I also see what looks like someone at 187.17.106.abc trying to use you as an email proxy. I don't know anything about the IP except that it's in Brazil.

[wilderness]

also require an adjustment in CP to designate your NEW custom-error-pages.

I did not get totally the last bit of your post.

Nearly all shared hosts use the same access interface.
Control Panel (i. e., CP)
Current Versions may very from host-to-host.
My hosts current version offers in the section titled Advanced

And in that section there is a sub-section titled Error Pages, where you edit the location of your Error Docs (i. e., remove their own custom Error Docs that include advertising.

FWIW, my past three hosts have used these advertising methods, and I'm assuming its the default setup for the CP Interface.

[qimqim]

Hi Lucy

There is also recurring refererence to mod_rbl and one ModSecurity. This may explain the 500 errors: If your host uses mod_rbl and/or mod_security to deny requests, it's their choice what error code to return. They might well choose "500".

Following your comment above, I sent a message to my Host (JustHost) asking to clarify and gort the following response which I can't understand:

On what url did you see 500 error? I went to mod security and whitelisted the rule. Can you retry please?

What do they mean by whitelisting the rule?

[wilderness]

What do they mean by whitelisting the rule?

I'm inclined to believe that justHost (the same one I use on multiple domains) doesn't have a clue as to the definition of white-listing!
Rather, what they meant to convey was comment-out (i. e, #).

FWIW and regarding same host?
A few months ago I sent them a trouble ticket for a friends website that I administer.
The ticket was the result of 30,000+ successive POST requests by the same IP and UA, which were denied by htacess configuration. (The allowing of 30,000+ POST requests was a vulnerability that effects all their customers and I believe they should be aware of same).
They response was a suggestion to add the IP into the deny from using their CP!

Absurd that they were even incapable of comprehending that the UA was already denied in the existing/active htacess that I configured for the same website!

With the above aside, JustHost is a very reliable host, however I've just learned to move aside any thought that they are capable of comprehending htacess and will not waste my time notifying them of any further server vulnerabilities that might come to my attention.

[qimqim]

Your error logs do contain some useful information, though, notably lines such as
RewriteCond: NoCase option for non-regex pattern '-d' is not supported and will be ignored.
Didn't I say something about [NC] earlier? Here fortunately it only means that the flag will be ignored, not the entire line-- or entire RewriteRule, or entire htaccess. But you're racking up an awful lot of this category of error, since the lines have to be read on every request.

Hi Lucy

I never got round to understand if I should delte the [NC]

meanwhile could you tell me how I can block a particulat IP address from accessing my site, through the htaccess?

Many thanks

[wilderness]

There's an active thread on the bottom of this page that provides examples.

IP Range Block may not be working? [webmasterworld.com]

[qimqim]

Hi

Thanks

Will this do?

#1 # block visitors referred from indicated domains

RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} make\-money\-online\.7makemoneyonline\.com [NC,OR]


RewriteCond %{HTTP_USER_AGENT} libwww-perl
RewriteRule .* – [F]

#1a block visitors from indicated IPs
order allow,deny
deny from 78.110.60.230.
allow from all

[wilderness]

Should you begin denying to the precise IP than you'll be adding IP's by the tens of thousands.

The procedure is to do a lookup (WHOIS) on the IP and explore the provider range.
In this instance 78.110.48.0 - 78.110.55.255 CIDR 78.110.48.0/24

change your deny from 78.110.48.0/24

or even deny from the entire Class A (RIPE-Euro) if you have no interest in traffic from the region. (Note this is a mere drop in the bucket for RIPE-Euro ranges.)

deny from 78.

[qimqim]

hi

The bandit in question keeps changing the domain but it all comes from 78.110.60.230

I don't understand why the range you mentioned. I would have thought the next step would be to deny his host.

Incidentally, I eem to have a dot at the end of the IP address. Should I take it out?

[wilderness]

I don't understand why the range you mentioned. I would have thought the next step would be to deny his host.

Incidentally, I eem to have a dot at the end of the IP address. Should I take it out?

The CIDR range I provided is the range of the host, BTW is a hosting server. Your bandit is a customer of same hosting service with multiple domains. (SEE the Server Farm Threads in the SSID forum).

No period after the Class D range.

[qimqim]

It's all getting very complicated for my remaining brain neurons and they are spinning....

So, what you say is that that bit of the .htaccess file should be

RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC,OR]
RewriteCond %{HTTP_REFERER} make\-money\-online\.7makemoneyonline\.com [NC,OR]


RewriteCond %{HTTP_USER_AGENT} libwww-perl
RewriteRule .* – [F]

#1a block visitors from indicated IPs
order allow,deny
deny from 78.110.48.0/24

allow from all

and that will block all traffic from the host of the original IP. Right?

[wilderness]

and that will block all traffic from the host of the original IP. Right?

Well sort of, but basically yes from that IP range.

Most hosts have multiple server ranges and that's likely for RU-HT-SYSTEMS as well.

[qimqim]

Hi

Thanks

I looked at WHOIS and am wodering where you got the end /24 from- I saw the 78.110.48.0 towards the end, but not the rest. What's the secret here?