More semalt variations

[blend27]

ip: 179.212.0.0 - 179.215.255.255
179.212.0.0/14
(Brasil)
Accept-Language: pt-BR,en,*
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
referer: http: //123456NN.responsive-test.net/

I've changed the italic part. So the visiting the original URL redirects to what-ever the actual value mapped to: http: //youresponsive.com/device?url=exampledomain.tld

it then tries to load a beacon to something.yadro.ru/?kjhsadfiuafherfoerf&semalt


So here we go:

#referer spam
RewriteCond %{HTTP_REFERER} (pizza|burger|button|for-your-|semalt|seo|--production|x00_|s-anal|responsive-test)
RewriteRule .* - [F]

more on semalt and such: [webmasterworld.com...]

The following A records are set to 217.23.9.249(youresponsive.com):
semaltmarket.com, uatraffic.com, youresponsive.com

inetnum: 217.23.9.0 - 217.23.9.255
netname: WORLDSTREAM

[edited by: keyplyr at 10:52 pm (utc) on Mar 15, 2016]
[edit reason] depersonalized IP address [/edit]

[lucy24]

Oh, yes, responsive-test. Took me several days to figure out that it's referer spam, not a bona fide testing site. Happily they all fail a basic header test, so I haven't needed to add any rules.

[aristotle]

---

System: The following 4 messages were cut out of thread at: https://www.webmasterworld.com/search_engine_spiders/4795332.htm [webmasterworld.com] by keyplyr - 7:11 pm on Mar 14, 2016 (UTC -8)

---

I've stared seeing this bot too. Strangely, another one called "responsive-test" began showing up recently as well.

 / 
 Http Code: 200 Date: Mar 13 14:05:37 Http Version: HTTP/1.1 Size in Bytes: 24049 
 Referer: http://11409774.responsive-test.net/ 
 Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36 

So one of them checks your site's up time and the other one checks its responsive design.

Maybe some person or group is doing a survey of the web for a research project.

[edited by: keyplyr at 10:54 pm (utc) on Mar 15, 2016]
[edit reason] depersonalized IP address [/edit]

[lucy24]

<topic drift>

unusual in that it fetches images and executes the statcounter code

That is unusual. Different kind of infection?

Mine-- I just rechecked logs and found only eight, though it felt like there's been more-- only gets the front page. Most botnets that I see will request supporting files --including, weirdly enough, embedded fonts-- but not images. And they hardly ever execute javascript-- I assume that's how your stat counter works-- except the ongoing inexplicable ones from Drake Holdings (the ones that dress like the plainclothes bingbot) which make the full piwik request.
</td>

[aristotle]

Well Lucy if you already had it blocked, you couldn't have seen what it does if it's allowed into the site.

[lucy24]

My 403 page includes a small image, a stylesheet and piwik code-- all accessible to everyone-- so I can tell when a human gets locked out. 99% of the time it's not really a human, though; it's an infected machine from Brazil or Russia*. They'll get the stylesheet, but not the image, and won't execute javascript.

Most importantly, the 403 page includes the regular header-logging code, so I know exactly who got locked out and what they wanted.


* Full-fledged robots live in Ukraine. Infected browsers live in Russia. Go figure.

[aristotle]

Lucy -- So just to clarify, are you saying that it didn't execute javascript or fetch the image on your 403 page? If that's the case, why did it act differently on my site?

[blend27]

Well, on the homepage hit(soft 403, cause of Brasil):

Full fledged Headers, took Images and executed JS.

The funny part is that it reported 100x30 screen res, at which point the IP was locked out for good.

[lucy24]

why did it act differently on my site?

Search me :) Maybe the whole question is a red herring: nothing to do with the putative referer, just the nature of the virus. I think all these referer-spam requests are made by infected human machines, so some of them really, really think they're browsers and act accordingly, while others follow a shorter script. I find it's especially rare for infected browsers to request the favicon; I've always assumed it's because the favicon is not explicitly named on my pages, and the script just says "follow all links".

soft 403

What does this mean?

[blend27]

What does this mean?

Its nice and fluffy :)

I don't let several countries into that particular website, and the message displayed to the user actually says so - Area Riservata: Accesso Negato.

[blend27]

Add free-video-tool.com to that spam referrer list. Just redirects to semalt.

#referrer spam
RewriteCond %{HTTP_REFERER} (pizza|burger|button|for-your-|semalt|seo|--production|x00_|s-anal|responsive|free-video-tool)
RewriteRule .* - [F]

[aristotle]

free-video-tool.com

This is another one that downloads images and executes the Statcounter code

[lucy24]

I found that there are two basic types of referer-spam robots. One is clearly robotic, with minimalist headers. The other sticks closer to the infected host's real headers, so for example "Accept-Language: pt-BR" (or ru-ru or whatever it may be, but Brazil is certainly prominent). Anything that triggers the statcounter is probably the second type.

:: idly wondering how many sites still have statcounters just because we never got around to deleting the code ::

[aristotle]

:: idly wondering how many sites still have statcounters just because we never got around to deleting the code ::

The isn't the old-fashioned site visitor counter. It's a full-featured traffic analysis package from statcounter.com. I only have the free version, but in my opinion it's much better tha google analytics. I've used it for at least 10 years, think it's great and highly recommend it.

[lucy24]

Ah. So I should be looking, similarly, for robots that not only download piwik.js but execute it. I kinda doubt they're intelligent enough to recognize the names of analytics programs, except when it's the kind of robot whose sole function is to bollux up your analytics. The rest of them probably just have instructions to execute any scripts they happen to meet.

If it's possible to do so, take a closer look at their output and see if it's truthful. (I noticed this detail recently while taking a closer look at the folks from Drake Holdings. In addition to professing a somewhat unlikely screen size, they consistently claim to have no referer ... even when they sent a Referer: header. They do send the correct time, which would also be easy to misrepresent.) Where do you draw the line between making things up for plausibility, and flat-out lying?

[aristotle]

The words "truthful" and "fake" don't harmonize very well

[blend27]

And another one...

http://keywords-monitoring-your-success.com/try.php?u=http://example.com

That domain was registered 2016-04-11 21:54:16

[lucy24]

I think some people just block "try.php" wherever it occurs.

[blend27]

Yes they do.

...and some just do ".php?u=" ... works for some for the past year or so.

[lucy24]

At one time I simply blocked any request for php (technically I served a manual 404), since I don't use it in URLs.

[webcentric]

At one time I simply blocked any request for php

Are you saying you don't do this anymore. I don't use it anywhere either. I have some specific php blocks like the one's mentioned above but I've wondered if I can just get away with just "php"

[lucy24]

The only reason I removed the line is that I've gone to a different system of robot-blocking, so the bots that would have requested bogus-url.php are now getting blocked anyway. In Apache, it looked like this:

RewriteCond %{THE_REQUEST} \.php
RewriteCond %{REQUEST_URI} !(real-url|other-real-url)
RewriteRule \.php - [R=404,NS]

The two elements %{THE_REQUEST} and [NS] are redundant, or what linguists call Double Markedness. Belt and suspenders. (Er, "belt and braces" for any passing Brits, unless you want a very odd mental picture. If you don't speak Apache: [NS] means "no subrequests", so the rule is ignored on things like includes or auto-indexing that really do use php, but only internally.)

The 404 response is exactly what they'd get if the rule didn't exist-- but returning it manually means the server doesn't have to go look. The visitor doesn't know how the 404 was generated, so they're not getting the "Uh-uh, we're onto you" information that a 403 would give.

I do the same on explicit requests for /includes/blahblah. This was my mistake, kind of, because if I'd thought of it in time I would have given the directory an entirely different name, not one that robots are likely to ask for anyway. (On my test site, every directory has some completely ridiculous name. It was fun making them up.) The only way to change it now would be to edit every ... single ... last ... page, which kinda defeats the purpose of having includes in the first place :)

[webcentric]

@lucy24 That makes perfect sense though I haven't a clue as to how to translate the NS feature to .NET, yet. So far, I've got a Request Filtering module working and the rewrite module. Next comes IPSecurity which can handle CIDRs, The 404s vs 403s is also desirable so, I'll have to look into how to control this in the configuration file as it relates to specific rules. Getting as much logic into the config as possible is my goal now so thanks for the pointers. I pretty much have to translate any htaccess concept into .Net so the explanation NS actually opens a door in my mind.

So, I just saw another of these obnoxious semalt-related requests (don't remember which one) the other day and it's what motivated me to look further here. I can't remember if all those domain variations contain the try.php string but I'm guessing they do and that, for now, seems like an excellent solution over adding these things one at a time as they come along.

[lucy24]

I should clarify for the benefit of non-Apache-speakers that %{THE_REQUEST} and [NS] are not actually synonymous; they just overlap.

Subrequest = anything the server does internally involving something other than mod_rewrite. Server-Side Includes are a good example; there's probably an exact IIS equivalent. Another is almost anything involving indexes. If you ask for /directory/ the server will send out /directory/index.html (or index.php or whatever); that's a subrequest. If the server has to auto-generate an index, that's yet another subrequest.

THE_REQUEST = whatever the user asked for. If they ask for /directory/short-pretty-url and you secretly rewrite to /some-other-directory/index.php?long-messy-query, that's not a subrequest. But it would fail a THE_REQUEST test, because the user didn't ask for index.php?blahblah. And if they ask for somepage.html and you won't let them see it, or it doesn't exist, the server now has to look for forbidden.html or missing.html or similar-- but THE_REQUEST is still the originally requested URL.

And finally, neither of those has much to do with the URL that a human user types in. Once you've said /directory/filename.html, your browser will dutifully ask for images and stylesheets and so on, and it will follow any redirects within reason without asking your permission. All of those count as requests even though your human wasn't aware of making them. (But they'd sure raise a stink if that stuff didn't get requested. If you've been to the Internet Archive in the last day or two, you can see what happens when there's a glitch and stylesheets don't get sent. Bleahh.)

[blend27]

...I can't remember if all those domain variations contain the try.php string but I'm guessing they do...

not all of them, but for the ones that do:

<rule name="blocking semalt referer spam" stopProcessing="true">
 <match url=".*" />
 <conditions>
 <add input="{HTTP_REFERER}" pattern="(try\.php\?u\=http)|(or\-something\-else)" />
 </conditions>
 <action type="AbortRequest" />
 </rule>

What I like about this "AbortRequest", it actually stops request all together, nothing is sent back to requester, 0 bites, thus no entry is generated in IIS log files!

And that is even better, cause that really defeats semalt junk. Simple, yet extremely Effective.

[blend27]

Another one:

http://fix-website-errors.com/try.php?u=http://mysite.tld

[wilderness]

blend,
Not sure if you saw lucy's mention?
There's not any need to keep expanding these refers in your htaccess.

All you need is try for their current and crawler for their earlier
RewriteCond %{HTTP_REFERER} (crawler|try)\.php

[blend27]

Don,

I only have one honeypot forum(not just honeypot, but entire forum acting as honeypot) left where I use .htaccess to trap these. On rest of sites, since they're hosted on IIS, I don't even see these(semalt) entries in log files due to web.config/native IIS url-rewrite-module configuration that I have outlined in a post prior to my last.

Cheers.

[aristotle]

here's one that showed up a few days ago, and is a real pest

http://seo-2-0.com/try.php?u=http://example.com

This could go on forever

[blend27]

I was watching sem~alt(Search Engine Marketing Alternative) for a while, I have contacted them with a simple questions once.

No Reply.

Well, I don't even know anymore, they just don't appear in my logs.

<action type="AbortRequest" />