Bots sharing info

Forum Moderators: open

Message Too Old, No Replies

Bots sharing info

roshaoar

10:06 am on Nov 18, 2014 (gmt 0)

Something I've noticed quite a bit in the last week or so is the extent to which many bot behaviours seem to be related, and I was wondering whether anyone else had some insight into this.

For example I get an insane amount of GET/POST/GET china-origin 'no referrer' stuff coming at a few selected, seemingly random pages. Having barred these through .htaccess I now find that some named bot 'TencenTraveler' now visits the exact same pages.

Likewise I stopped AhrefsBot using .htaccess. 5 other bots have now started appearing out of nowhere, and one odd buttons-for-website dot com referrer, all using the Brazilian and French IPs, like AhrefsBot was using.

So I wonder if these guys sell data to each other, or have a thing whereby one crawler detects it is blocked it puts your domain in some other bot's queue to still get at sites. This might make business sense to the owners of these as they're trying to datafy as much of the web as possible?

Anyone?

Cheers

keyplyr

6:43 pm on Nov 18, 2014 (gmt 0)

Common software or server vulnerabilities are sought by quite a few intruders but probably unrelated to each other.

Some vulnerability attempts come from infected machines & user ISP accounts. These may or may not be from the same bad actor.

There are actually places on the internet where anyone can buy/sell info, so that may also affect what you are seeing.

not2easy

6:54 pm on Nov 18, 2014 (gmt 0)

If you are blocking by UA and the UA is hitting 403s they just change the program. Referrers are frequently homemade and so are UAs. Only the IP tells you who/what was there.

lucy24

7:30 pm on Nov 18, 2014 (gmt 0)

For example I get an insane amount of GET/POST/GET china-origin 'no referrer' stuff coming at a few selected, seemingly random pages.

Low-budget botnets. At any given time I have from 2-4 botnets that I can identify after the fact by behavior alone. They must buy lists from somewhere, but it's highly unlikely you will ever be able to identify the seller.

Currently, for example:
-- "contact" botnet: some random page giving my root as referer, followed by contact page giving the previously requested page (regardless of response) as referer
-- "index.php" botnet: complicated but identifiable pattern ending with a set of requests for example.com/index.php
-- "nyet.gif" botnet: requests include "PUT /nyet.gif" followed by "GET /nyet.gif" (again, even though PUT never succeeds)

Some of the above requests will get an automatic 403. In all cases I can check the IP after the fact and, if it turns out to be a server, colo or similar, block it that way. It's your call whether you choose to block human ranges when they're from a country that doesn't send you measurable human traffic. In general it's not worth blocking an individual human IP from your own country.

roshaoar

8:52 pm on Nov 18, 2014 (gmt 0)

This is why I posed the question though, because these bots seem to achieve relatively little. So I thought maybe the real value was in selling their trawl data...

keyplyr

9:56 pm on Nov 18, 2014 (gmt 0)

I block China :)

lucy24

2:26 am on Nov 24, 2014 (gmt 0)

buttons-for-website.com

I'd never seen this referer before, but thanks to this thread (and another of similar vintage) it jumped right out at me the last time I processed logs. Happily the requests were already blocked, thanks to a header quirk that someone else recently pointed out in yet another thread. Pattern of requests suggests that these, too, are coming from infected human browsers, mainly but not exclusively in Brazil.

dstiles

8:39 pm on Nov 24, 2014 (gmt 0)

From a zdnet rss feed:

"Online cybercrime 'schools' and a vast array of products and services focused on those wishing to enter the life of digital crime are more easily available in Brazil than any other country, says study."

Which is ironic, considering brazil is foremost in trying to improve internet protocols.

I do a "soft" block on a few countries including brazil, china, ukraine etc but I'm seeing a huge increase of bad hits from china this month, almost all aimed at one innocuous little site that's of no serious import to anyone, just a small local business. Not just china - its being hit by rss seekers at up to 1000 a time for some IPs, probably due to a fault in the rss seeker (I wrote to magpie and was ignored). I suspect the site has been registered somewhere nasty by a competitor (the site owner had a trolling a few years back) but tracking that down is pretty much impossible.

Odd, but a high percentage of the chinese IPs are "mobile". Possibly, I suppose, their phones are readily compromised into botnets.

I blocked the referer buttons-for-website for referer spam as part of the musica blitz - mostly brazil IPs but a few european ones as well.

keyplyr

10:54 pm on Nov 24, 2014 (gmt 0)

"Online cybercrime 'schools'... are more easily available in Brazil than any other country, says study."

If the school is "online" then why would it matter what country it is "available" from?

dstiles

8:31 pm on Nov 25, 2014 (gmt 0)

Not sure if it refers to "real" schools teaching cybercrime or online schools. Either way, I suspect the item was not too carefully written and should indicate that the home of the schools is in Brazil. And possibly the language is not English, which could localise the pupils to Spanish speakers and Brazil is certainly a major cybercrime source.

roshaoar

8:51 pm on Nov 25, 2014 (gmt 0)

I wouldn't be too quick to point a finger at just Brazil. Just as my semalt crawl referrals have now started coming from various countries, now too these things are as well. It's weird, too much of a coincidence.

ex141028.log(21637): 188.218.22.xxx - - [28/Oct/2014:16:00:12 +0000] "GET / HTTP/1.0" 200 13678 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141029.log(21275): 187.104.17.xx - - [29/Oct/2014:17:12:08 +0000] "GET / HTTP/1.0" 200 13721 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141030.log(20819): 87.110.40.xxx - - [30/Oct/2014:13:35:37 +0000] "GET / HTTP/1.0" 200 13690 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141030.log(42225): 177.68.128.xxx - - [30/Oct/2014:23:36:05 +0000] "GET / HTTP/1.0" 200 13670 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141102.log(3332): 125.25.184.xxx - - [02/Nov/2014:03:39:59 +0000] "GET / HTTP/1.0" 200 13691 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141110.log(1004): 37.61.58.xxx - - [10/Nov/2014:01:15:43 +0000] "GET / HTTP/1.0" 200 13773 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141113.log(20410): 85.232.220.xxx - - [13/Nov/2014:19:13:02 +0000] "GET / HTTP/1.0" 200 13756 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141115.log(6865): 36.78.162.xx - - [15/Nov/2014:06:07:34 +0000] "GET / HTTP/1.0" 200 13807 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141117.log(10317): 87.1.38.xx - - [17/Nov/2014:13:18:37 +0000] "GET / HTTP/1.0" 200 13706 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141117.log(27892): 189.26.55.xx - - [17/Nov/2014:22:48:12 +0000] "GET / HTTP/1.0" 200 13793 "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141118.log(24057): 187.73.48.xx - - [18/Nov/2014:17:29:51 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141119.log(11175): 41.182.173.xxx - - [19/Nov/2014:12:03:07 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141120.log(18552): 186.251.219.x - - [20/Nov/2014:16:24:49 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141120.log(23857): 179.97.82.xx - - [20/Nov/2014:20:03:38 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141121.log(10801): 105.107.113.xx - - [21/Nov/2014:10:49:35 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141123.log(14540): 95.93.39.xxx - - [23/Nov/2014:13:38:44 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141124.log(3158): 179.213.109.xxx - - [24/Nov/2014:02:39:46 +0000] "GET / HTTP/1.0" 500 - "http://buttons-for-website.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
Found 24 occurrence(s) in 17 file(s)

ex141121.log(288): 200.102.118.xx - - [21/Nov/2014:00:31:04 +0000] "GET / HTTP/1.0" 500 - "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141121.log(18377): 201.70.137.xxx - - [21/Nov/2014:15:21:57 +0000] "GET / HTTP/1.0" 500 - "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
ex141122.log(10849): 84.124.139.xx - - [22/Nov/2014:12:26:11 +0000] "GET / HTTP/1.0" 302 195 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141122.log(16780): 189.15.180.xx - - [22/Nov/2014:17:17:40 +0000] "GET / HTTP/1.0" 302 195 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
ex141123.log(10492): 200.199.169.xx - - [23/Nov/2014:10:28:21 +0000] "GET / HTTP/1.0" 302 195 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
ex141123.log(17631): 95.155.208.xxx - - [23/Nov/2014:16:27:58 +0000] "GET / HTTP/1.0" 302 195 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141123.log(20260): 89.21.213.xxx - - [23/Nov/2014:17:59:14 +0000] "GET / HTTP/1.0" 301 202 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141123.log(25450): 177.16.73.xxx - - [23/Nov/2014:21:06:04 +0000] "GET / HTTP/1.0" 301 202 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141123.log(27877): 190.193.66.xx - - [23/Nov/2014:23:09:02 +0000] "GET / HTTP/1.0" 301 202 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141124.log(587): 177.103.77.xxx - - [24/Nov/2014:00:15:55 +0000] "GET / HTTP/1.0" 301 202 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
ex141124.log(11668): 179.111.206.xx - - [24/Nov/2014:11:29:39 +0000] "GET / HTTP/1.0" 301 202 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
ex141124.log(17851): 189.25.221.xxx - - [24/Nov/2014:14:31:32 +0000] "GET / HTTP/1.0" 301 202 "http://semalt.semalt.com/crawler.php?u=http://xxxxxxxxxxx.xx.xx" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"

Leosghost

8:51 pm on Nov 25, 2014 (gmt 0)

After my very 1st site was hacked in 98 ( home page defacement )..host was also hosting IRC chat rooms, HC pron, hacker fora etc..They "got in" via another site on the server..Host knew nothing about security, but liked money..

We ( I and the person who taught me how to build that site, not me and the host ) traced the hacker group to Brazil, found them in a forum, giving instructions on how to hack sites ( via in that particular "teaching" case, php exploits ) to their newer members..

lucy24

9:15 pm on Nov 25, 2014 (gmt 0)

the home of the schools is in Brazil. And possibly the language is not English, which could localise the pupils to Spanish speakers

Uh...?

Leosghost

10:01 pm on Nov 25, 2014 (gmt 0)

the home of the schools is in Brazil. And possibly the language is not English, which could localise the pupils to Spanish speakers

Uh...?

Well caught lucy24 :))..I missed that ..yes indeed.. Uh ? ;)

keyplyr

10:51 pm on Nov 25, 2014 (gmt 0)

Well Portuguese kinda sounds a little like Spanish with a little French mixed in, somtimes if the light is just right :)

lucy24

8:18 am on Nov 26, 2014 (gmt 0)

To me Portuguese sounds like Arabic. It looks like what it is, a Romance language, it just doesn't sound that way to my ears. But then, most languages I don't know sound like Arabic to me.

roshaoar

5:15 pm on Nov 26, 2014 (gmt 0)

Looks like I was kind of right about this, they're related. Just google semalt, buttons, botnet and soundfrost. Ugly stuff

dstiles

9:06 pm on Nov 26, 2014 (gmt 0)

Ok, Spanish, Portugese - neither is exactly like Brazilian but English it ain't. :)

roshaoar:

buttons uses the same (or very similar) attack method as semalt. I have several domains blocked for using that method. It was mentioned hereabouts that semalt-style attacks may have originated or been controlled by BR and UA but I've seen ES and a few other countries involved. If it's like other botnet activities it's quite possible the software is rented out by its originator to anyone who wants it.

roshaoar

9:16 pm on Nov 26, 2014 (gmt 0)

Yes, that correlates with what I've been observing - and not just that, but also the same ISP as Semalt. It's been added to Joram van den Boezem's Semalt blocker library on github now as well - all good stuff