back in the Ukraine

if the User Agent fits, wear it

[lucy24]

Oi! Where'd Search go?

I've mentioned my Ukrainian pals before but never got around to asking if other folks have met them. Forums Search (via g###) is silent.

Most of the time they live in the 193.106.136-139 range. (Their actual addresses so far are 193.106.136.33-58 but, hey, easier to block the whole sector.) Sometimes they wander over to 92.249.0-127 (they're fond of a specific address in 127, but again, easier to nuke the whole thing) and once showed up at an address in 95.132-135.

What makes them absolutely distinctive is their behavior. Each visit goes like this:

First pick up a forged referer from .ru, .ua or .su. (All of these are blocked, with an exemption for google.ru. Also .mobi, for some reason.) Put on a fake UA. Try for a specific interior page,* and then the front page. Change user agents, keeping the same referrer, and repeat. Same again, for a total of 3-5 attempts. Rarely more, usually just three. So each individual visit shows up as 6 or 10 entries in the Error Log. (It used to be 9 or 15 until I unblocked the 403 page. I don't know why the server asks for this after requests for interior pages but not for the index page.)

Later in the day, they beg, borrow or steal a fresh referrer, and come back with a new set of 3-5 rented costumes.

When it hit seven visits in one day I caved in and changed from 403 to 127.0.0.1. This has not reduced the overall number of visits, but has cut each one from 6-10 hits to one. I guess they are so flummoxed at seeing their own house that they forget about changing clothes and pounding on my door again.


* For a long time they focused their attentions on "Know Your Lion", but a while back they shifted to "Alonzo and Melissa". It is, of course, pure coincidence that the shift came at about the same time that g### got around to keyword-indexing the latter. Both of these pages are open to robots, but the Ukrainians are not in a position to know this, having never looked at robots.txt.

[wilderness]

denying the following will generally solve those issues ;)
1(0[0-9]|1[0-9]|2[0-6])
19[3-6]
2(1[789]|2[01])
7[789]
8([0-9])
9([0-5])

I've hordes more.

[lucy24]

100 through 126? Ouch! I personally know* someone at 109.nnn. :) Looks like there's a UK "pocket" at 109.144-159. Uhm, come to think of it I know someone at 78.nnn too :: snicker ::

Hm, lessee who else. Yup, there's a Norwegian at 84.202, a fair number of Italians in the 70's and 80's including another 78... Better not touch that area.

I could probably do a blanket ban on Eastern Europe with no ill effects. Can only think of one g### query that came through in Cyrillic, and it didn't result in a hotlink so I must not have had what they were looking for ;)

* * *
Had a visit from an absolute jaw-dropper yesterday. Mainly lived at 192.nnn, but got a visiting friend from 80.nnn to help out at the end. Left me so flabbergasted, I saved the logs in a separate file.

#1 At first glance a typical robot hit, following all links systematically-- they even gave each referrer, making it easy to keep track-- including any images linked in <a href> form. Total, 200 pages (exactly!) and 142 jpgs. They didn't ask for no steenking robots.txt, so they got about 30 more pages than they needed. Ahem.

#2 Along the way, they attempted to pick up 181 style sheets. If you're thinking this is a lot of external css for 200 pages, you are right: I have a grand total of nine. (I counted.) But it requested each style sheet all over again for each page. I say "attempted to pick up" because they laid aside their UA for each css request, resulting in an automatic 403.

#3 After this part-- which took about 15 minutes, at a respectable spacing of 2-5 seconds-- they rested for about an hour. Then they took their clothes off-- meaning that everything from here on got 403'd at the gate-- and came back for all the regular images with <img src> links.

#4 I guess taking all those pictures-- or getting all those 403s slammed in your face-- was tiring work, because about 1/6 of it was done by new arrival 80.nnn, also not wearing any UA. No duplications, no omissions except the background images. I eventually figured out this was because they couldn't get the css and therefore didn't know the backgrounds existed. They also didn't pick up the external js, which is named within html files.

MYSTERY. I have noticed the following a couple of times before with high-volume unwanted robots. They went for absolutely everything-- except the ebooks directory, which is linked in exactly the same way as all the others. Robots can't read, can they? Are they afraid they will be asked to read? What on earth is keeping them out of there?


* No, I don't go around asking people's IP addresses. ("You're a 192? Sorry, my numerologist says we're not compatible.") It's from one of my font-and-whatnot experiments grabbing volunteers from just about everywhere. Conclusion: MSIE 8 stinks at both Naming and Substitution, and Iceweasel gives false positives in the javascript/css test.

[dstiles]

The basic rule of anti-scraping is to block ALL server farms. That includes 193.106.136/22 and a LOT of other Ukrainian ranges. For some sites I block all Ukrainian IPs even if they are not servers (at least, IP ranges I know about).

Once you have blocked server farms, look at the browser/robot header fields and analyse them carefully - including, of course, the UA.

If you do not block scrapers then there is a fair chance your site lose its importance in SEs.

[lucy24]

That includes 193.106.136.0/22

Yup. Got 'em blocked in those identical words. (136-139, smileweb, not ideally named ;)) I'm only aware of their presence because I have to look at the error log now and then. If it's unusually fat I worry that I goofed in an htaccess revision and it's throwing 500's all over the place. That's how I learned about the blocked requests for the 403 page; it doesn't show up in the access log. And once in a blue moon they borrow someone else's computer and hit from there, which is why I put in the supplementary block on Ukrainian referers.

countryblock.net goes on for miles, but they're saying things like
deny from 2.56.0.0/16
deny from 2.57.0.0/16
deny from 2.58.0.0/15
which can obviously be compressed into a single statement ending in /14.

Oddly I've yet to meet a robot that forges the IP and referer and UA concurrently. (I'm small enough that the pattern of activity would still jump out at me.) It's just one or two out of the three.

Once you have blocked server farms, look at the browser/robot header fields and analyse them carefully - including, of course, the UA.

That's not very hard when they're sending blank UAs :) As does google's ### faviconbot, which may explain the occasional download by an apparent human of nothing but the favicon. Everything else is cached.

But I still wish I knew what makes certain pages or directories especially attractive or unattractive to robots. (What makes them attractive or unattractive to humans is of course a whole nother question for a whole nother subforum :))

[walkman]

smileweb has hit me several times. Totally crashed the server.

[dstiles]

It is very difficult (but not impossible) to forge an IP. More likely is a bot accessing via a proxy. This can usually be traced and blocked if it's not an approved proxy. I often find a server farm IP trying to access via a botnet IP that's been configured as a proxy, for example.

[g1smd]

There's a fair number of proxies built using Google AppEngine. Most don't have a robots.txt file, so the whole web gets indexed as Duplicate Content, prefixed with the proxy domain name. You'd have thunk Google was clever enough to know what was going on there, or would enforce a robots.txt file to stop that happening.

[lucy24]

It is very difficult (but not impossible) to forge an IP. More likely is a bot accessing via a proxy.

I've only had one that caught my attention, and it was way back in February so I don't have the raw logs. They dressed up as (or actually were)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
and grabbed a total of eighty pages. (Since I don't have the raw logs, I don't know if they also got images.)

Matter of fact, I'm still leery of the NET CLR 1.1.4322 element, though it shows up all over the place.

:: shuffling papers ::

Oh, what the hey. I had the text editor sort them into numerical order in case anything might jump out at anyone. The 220.244.34.43 duplicates were consecutive; the others I only noticed after I sorted. They must have liked 195.53.203.127, since they came back to it five times. Curious timing, now that I look at it more closely. Anywhere from two in one second to several minutes apart. The whole thing covered about 40 minutes.

41.0.29.66
41.134.81.234
41.240.56.50
58.240.237.32
63.146.198.4
64.202.101.154
66.232.167.19
67.23.234.165
72.4.71.66
74.241.225.81
76.73.94.222
77.42.159.218
77.237.178.169
78.155.120.77
86.114.69.67
88.149.218.58
89.76.221.151
91.135.84.122
94.23.193.213
95.56.229.34
110.93.196.11
110.232.82.2
114.31.3.10
116.90.208.139
118.96.150.93
119.1.174.28
120.50.62.51
120.50.177.98
120.88.8.58
121.52.72.136
122.154.140.82
123.242.153.99
125.162.243.42
145.222.84.71
161.67.130.202
178.19.21.195
178.173.128.11
180.246.120.4
183.182.84.8
184.106.150.219
187.8.236.91
187.115.202.171
188.165.214.5
189.19.249.109
189.22.105.204
189.135.54.163
190.41.180.147
193.137.203.231
195.53.203.127
195.53.203.127
195.53.203.127
195.53.203.127
195.53.203.127
200.88.113.147
200.96.27.200
200.123.69.211
200.162.160.208
200.174.158.18
200.185.148.137
201.9.109.181
201.39.242.194
201.40.43.139
201.82.23.220
201.168.20.134
201.251.63.124
202.43.181.130
202.143.143.130
202.143.191.2
202.143.191.2
203.189.146.217
207.252.1.194
208.69.72.99
212.48.121.144
212.199.164.168
213.251.187.190
216.58.41.195
217.31.36.132
220.227.90.238
220.244.34.43
220.244.34.43

At least they got what they came for. I had a brief visit a day or two back by a robot that stuck "/" onto the front of every link it met. This worked fine in the links from the top-level index file, but got progressively worse, including half a dozen very improbable "/../blahblah/index.html". They gave up after about 20 tries. Maybe they got tired of rats.

[g1smd]

Some of those numbers look a bit familiar having been blocked and dumped a week or three ago on several sites.

[blend27]

smileweb has hit me several times. Totally crashed the server.

Same here, had to ask host to block 193.106.136.0/22 at Firewall Level. Mad log spamming from that range.

[dstiles]

Lucy - if you got that many IPs hitting a single site I would suspect a botnet. They do not use a discernable proxy, although the botnet itself acts as one.

What I would not accept is that the IPs were forged. They really were those IPs you recorded, with an infected computer at the other end of each one.

One of the helpful things in following up each botnet IP is that several of them resolve to server farms, which can then be added as blocked ranges. If you really do not want Asian or Russian-bloc IPs then you can add those at the same time.

[lucy24]

One of the helpful things in following up each botnet IP is that several of them resolve to server farms, which can then be added as blocked ranges.

Uhm. Can someone point me to a place that explains in words of two syllables what's the difference between a server farm and shared hosting? I would prefer not to lock myself out-- or to be perceived as an Undesirable Link-- just because my site's got a floating IP and my host moves me to a different server every year or so.

If you really do not want Asian or Russian-bloc IPs

Your fingers automatically typed "Soviet bloc" and then remembered what year we're in ;)

then you can add those at the same time.

Well that's the moral question innit. Do I punish the honest Internet-users because they have the bad luck to live in a country that also has a lot of dishonest ones?

If the country in question is China the answer is: Oh, yeah, you betcha. No compunctions there. And my latest Stupid Robot-- the one who put / in front of everything-- was from Belarus. Now there's a country I don't at all mind blocking.

But there's also a surprising lot of hinky stuff coming out of Germany, and I really don't think I can lock them out.

Wonder if all those Chinese robots would go away faster if instead of blocking them I redirected them to a page that said in full "Free Tibet!" or similar. So far, redirecting my Ukrainians to 127.0.0.1 has had no effect on their number of visits, but it does mean that each visit is only one hit instead of six or more.

And they're still focused on Target Page #2. Far as I can tell, they've never even been to the intervening page (both of their targets are inner pages, so there's at least one step between it and the front page), unless they borrowed someone else's robot. Did they perform some arcane algorithm telling them that this page would be the best choice for whatever nefarious business they're planning? Their two targets have been different in just about every way I can think of: size, frequency of human visits, location, number of linked images, presence or absence of external css, distribution of keywords...

For those who are slow on the uptake: Yes, I am fascinated by robot behavior. What goes on in their robotic brains? How do they decide where to go and what to do? Does rudeness to a robot ever have any effect? Where do they get their marching orders?

[dstiles]

Server farm, in my book, means any block of IPs that contain servers. If I get really annoyed I may include the odd static DSL range but in that case it's usually Russian bloc - I did actually type that the first time and never thought soviet, honest! :)

It does not usually matter if you block your own server's IP, unless you have tools on the server that refer back to it using http protocol.

Some web sites are not relevant to (eg) non-UK DSL users. I block, for example, most of the world to a UK-only shopping directory because I'm fed up rejecting idiots from Ukraine, Germany, USA, India etc who cannot read the large UK Only sign on the submission form. Automatic form submission is usually taken care of by blocking the source server farms.

I agree Germany can be the source of a lot of bots, but with a couple of exceptions they all come, in my experience, from blockable server farms.

Again in my experience: China is very difficult to determine server from DSL. I do find, however, far more unwanted bots coming from USA than from China. India is usually more of a DSL SEO nightmare than a bot source.

The ultimate problem comes in ascertaining whether an IP range is a server farm or a DSL ISP.

Very few nasty bots take hints about leaving a site alone. Most return year in year out, ignoring robots.txt and 403's alike.

Robotic brains? I'm afraid Asimov's Positronic Brain has never been realised, especially in the case of internet robots. We're stuck with stupid bots that can barely (and often never) sign their own name in their UA. :)

[bdzpete]

Hi Lucy,

Thanks for your explicit description of the access pattern from the Ukranian visitors. It exactly matches my experience since they first visited me about 2 weeks ago:
- IPs 193.106.136.54|58 and 92.249.127.111
- visits always in groups of 3 in the same second
- about 6 groups of visits per day from the first IP; perhaps 3 from the other.
- undeterred by htacces-enforced 403 or php-enforced 404
- wide range of UA
- HTTP_REFERER a variety of .ru domains

In the last few days they've gone more global with their referers. In addition to the .ru URLs there are now URLs that appear to relate to:
adult .com websites
skype downloads from a .com website
seo from a .net website
mp3 from a .net website
health stuff from a .com website

I've also noticed almost-daily visits from 66.55.138.243 with an http_referer that includes trafficfaker.com . Are these visits somehow related to the Ukraine visitors, or could they signal the start of something more devious?

Pete

[Pfui]

bdzpete (welcome!), and lucy --

What you're seeing is all 'routine' botnet stuff. Been around for ages, will be around for ages. Machines from Kamchatka to Kansas to Kauai are compromised, infected, and thus dumb relays for loads of bad stuff.

If only everybody ran an up-to-date antiviral program at least once a day on every machine they controlled!

Anyway. What can you do to secure your stuff? Here's an off-the-top-of-my-head list o' tips. This forum's Charter docs and most threads contain more.

Block by IP/HOST:

The next time you spot a bad or even iffy IP, check it out at Project Honey Pot. Look at the data, and particularly note the "Spider Last Seen" data to double-check that the IP you're seeing is still a mess -- because they usually are. For example:

For 193.106.136.54 [projecthoneypot.org...]
For 92.249.127.111 [projecthoneypot.org...]

Scroll down any Project Honey Pot page and you'll usually see loads of interesting bits. For example, the "IPs In The Neighborhood" column is really useful when you're deciding whether to block a single IP or a range. (When it comes to former Soviet countries and China, I opt for a larger block without a second thought. (Google Russia botnet, or China botnet. 'Nuff said.)

Block by REFERER:

Look over the fake adult/pharma/health/etc. refs in your logs, pick the common words and block 'em. Look over the refs you know wouldn't contain a link to you, pick the common words and block 'em. (RESIST the temptation to go to those sites and check!) Look over the key words in the e-mail spam you get and -- wait for it -- block 'em.

Block by URI:

The vaaaaaaaaast majority of the botnet hits I get are PHP exploits. I don't run PHP so any URI with "php" in it gets 403'd. If you do, make sure ALL your programs are ALWAYS current; apply patches pronto. If hits are for PHP programs you don't run, block those words/names.

Block by UA:

One word: Whitelist. (Search this forum for same; particularly posts by mod IncrediBILL.)

Block by REQUIREMENTS:

Spambots make a beeline for pages with words/titles containing words like contact, mail, etc., ditto scripts containing the word mail. If you can't re-title those pages (and turn the original names into bot bait), make sure no one can POST to a script without the referer being its form/page on your server.

Block by RELAXING:

Bot-spotting, and bot-researching, and bot-related reading -- e.g. [blogs.forbes.com...] -- can be interesting, fascinating, and even fun. It's also a free time sucker-upper and a slippery slope if you have even the slightest OCD tendencies. So pick your 'battles' -- try not to focus on the minutiae of any specific botnet because it ain't worth it because botrunners are ever-present and pervasive. Instead look for botnet hit commonalities, make sure they're handled correctly (via Deny from, or Rewrite rules, etc.), and move on.

Last but not least...

Every now and then, check your own IP address at Project Honey Pot because your machine may be a botnet conscript just like all the machines you see.

FINIS

(Really:)

[lucy24]

Some web sites are not relevant to (eg) non-UK DSL users. I block, for example, most of the world to a UK-only shopping directory because I'm fed up rejecting idiots from Ukraine, Germany, USA, India etc who cannot read the large UK Only sign on the submission form.

Humans can be equally incomprehensible. If a download is explicitly labeled For Intel Mac, what do they plan on doing with it on their Windows machine? (And if they've got a utility that converts this particular file, I wish they would tell me about it because I would love to link to it ;))

I've also noticed almost-daily visits from 66.55.138.243 with an http_referer that includes trafficfaker.com . Are these visits somehow related to the Ukraine visitors, or could they signal the start of something more devious?

Far as I can tell, trafficfaker is a legitimate (in the sense of "it really exists") entity whose object is to persuade you to pay them. They annoy me. They used to live at 112.something and apparently moved at the beginning of the month. I just changed my htaccess to block by referer.

The vaaaaaaaaast majority of the botnet hits I get are PHP exploits. I don't run PHP so any URI with "php" in it gets 403'd.

Yes, someone around here-- possibly you-- suggested this earlier so I've now got .php globally blocked. I don't know if there's a difference at their end between 403 and the 404 they would otherwise get, but it does mean if I don't feel like investigating I don't have to go in and figure out where the 404 is coming from. (I did a bunch of rearranging about half a year ago and didn't know what a loooong memory search engines have, so I'm still mopping up. Most of them have at least been pinned down to 410. The real 404s, that is.)

[g1smd]

Ya know you can serve 503, 500 or any other number you care to experiment with in the HTTP header?

If you use URL rewriting and extensionless URLs you can block all requests with extensions unless they are for CSS, JS and image files or for ZIP files.

If your internal rewrite look like:

RewriteRule (.*) /script.php?param=$1 [L]

then you're doing it all wrong.

That is,

(.*)

is the wrong RegEx pattern here.

[dstiles]

66.55.128/19 is Choopa server farm and should be blocked.

Lucy: I doubt many humans download anything they can't use. Most scrapes are probably bots in some form, some running from servers (infected or deliberate), others from infected home/business machines.

[lucy24]

66.55.128.0/19

:: counting on fingers ::
Only 128 through 159? Looking up a random IP I get 128-255 which would be /17

I doubt many humans download anything they can't use.

Well, there was no question about my Romanian pals when they downloaded patches for Mac Classic games that they couldn't possibly own, or the Ukrainians' do-or-die quest for the e-book Rambles of a Rat-- which is available from other sources-- to say nothing of the bingbot's morbid fascination with my robots.txt file :)

But some patterns of behavior are definitely human. Main page from search engine, accompanied by external css, favicon and so on; suitable time lapse to read page; download of linked file with main page listed as referer. Maybe there is a utility that converts Mac OS X keyboard layouts into Windows format. Maybe they're downloading for a friend who's on dialup. Who knows ::shrug:: Most of them come in through google.it, so maybe they're thinking "It probably won't help but it can't hurt to take a look and it's only eleven K".

And let's not talk about how many separate times I have absent-mindedly downloaded that ### google pdf, forgetting that (a) it's a pdf and (b) I have already looked at it.

[dstiles]

160 and above are not choopa. In fact 160/19 appears to be DSL but I could be wrong. Either way, the choopa block is the only one in that /16 that I have blocked - ie that has given trouble.

I suspect the Ukrainian scrapes you are getting is not human browsing but bots/botnets.

Not everything that comes from google SERPS is human but I take your point. Whether it's a problem would seem to depend on the number of downloads? If it's only for MAC then bar MSIE and other Windows UAs - although firefox and one or two others are perfectly ok on MAC (and safer than safari). It's easy enough to detect the UA and not show the link on the page.

[lucy24]

I suspect the Ukrainian scrapes you are getting is not human browsing but bots/botnets.

Oh, I never had the least doubt about them. Humans don't keep a roomful of elderly computers with prehistoric versions of MSIE, all set to download the same page over and over without accompanying css and images but with forged referer ;) And humans don't keep coming back when they get the door slammed in their face eight times a day. Well, maybe Ukrainian humans do. Haven't met enough in person to tell.

Not everything that comes from google SERPS is human but I take your point. Whether it's a problem would seem to depend on the number of downloads? If it's only for MAC then bar MSIE and other Windows UAs - although firefox and one or two others are perfectly ok on MAC (and safer than safari). It's easy enough to detect the UA and not show the link on the page.

Not a problem at all. In fact, last time I looked at the logs I had a "cold" visit to the same page (not the download, just the lead-in page), this time from a Mac user, but again in Italy. I can just picture someone wandering down the hall and telling his colleague "Hey, I found a site that might have what you're looking for".

Maybe I should make a version of the page in Italian ;) One recent search was "{such-and-such} per word" and I had to do some serious head-scratching before I realized they were searching in Italian for {such-and-such} for (per) MS Word.

* * *
I just locked out FairShare via BrowserMatch. They only come around every month or two but they annoy me no end. If anyone seriously thinks anything in the /hovercraft/ directory is scraped, they're either delusional or we have an interest in common.

[dstiles]

Browsers CAN rotate both UAs and Referers but generally you are correct. :)

Did not know about fairshare but a quick search suggests it was from the Amazon Cloud. If that is so I would not have noticed it: all AWS ranges are blocked, some with extreme prejudice.