Now I know what is behind these Kaspersky hits

The Cerebro bot

[SumGuy]

Starting in Nov / 2020 I began seeing a pattern of hits from these IP's:

77.74.177.113
77.74.177.114
77.74.177.119
93.159.230.28
93.159.230.83
93.159.230.84
93.159.230.85
93.159.230.87
93.159.230.88
93.159.230.89

The host-hames for these IP's are cerebro-sdc-prod-10x.kaspersky-labs.com (where 10x runs from 100 to 109). The IP's belong to "Kaspersky Lab Switzerland GmbH". The hits only ever request default.html or index.html. No other file, including robots.txt, is ever requested. The User-Agent is always:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

During 2021 the hits were sparse, but started to ramp up in August. During the entire year 2021 there were only about 52 hits. Usually there are 3 or 4 hits on the same day. By Dec 2021 there were 13 hits that month, 3 hits per day on 3 of those days.

During 2022 there have now been 335 hits, 41 so far this month (November), and just today there have been 8 hits.

On any day where there are multiple hits, there will usually only be 1 hit from 77.74.177.119 and the rest will be from 93.159.230.0/24. The timing between hits is very close to 60 minutes, give or take 3 or 4 minutes.

Kaspersky is known for anti-virus software and god knows how many other security-related products. I would really like to know which of their products or services is responsible for these hits, what is the objective of these hits, and are they the result of a totally automated process that was, or was not, initiated by human or user direction or control.

I can imagine several scenarios, such as

a) website vulnerability or threat analysis (automated or user-initiated?)
b) website availability / down-detector?
c) website change detection? (ie has the site landing page changed?)

Does Kaspersky offer users a website change or down detector service? Can a user request that Kaspersky monitor a website (that they don't necessarily own or operate or have any administrative connection with) and Kaspersky periodically monitors the site and informs them of status changes?

Clearly I haven't interacted with any Kaspersky software or service to implement any sort of monitoring or surveillance of my site, but my best guess is that *someone* has - if indeed that is what is behind this web activity.

Thoughts? Does anyone else see what certainly looks like site-monitoring activity from these Kaspersky IP's?

==================

Edit : Just before I posted this, I did a quick search for Cerebro and Kaspersky. This is what I found:

------------------------
And one of our new robots, called CEREBRO, surfs the Internet all day and night scanning for malicious code, and when it finds some, it automatically categorizes it based on its content and sends the info on it to KSN, where other robots – also automatically – can apply protection for you. CEREBRO, a robot-crawler, has already scanned around a hundred million webpages (and doesn’t forget to keep rechecking them, just in case something’s changed).

[eugene.kaspersky.com...]

[lucy24]

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36 

That seems to be a popular robotic UA, though possibly less so than in years past, since it's pretty severely out of date.

I checked two ways: first for Chrome/71.0.3578.98 and then for the whole string. All but a handful--a literal handful, less than ten--of the short version also showed up as the long version. And about a fifth of those are from 77.74.230 or 93.159.230, all blocked due to a consistent set of header deficits. (Conversely, only one from those two /24 ranges used a different UA and wasn’t blocked.)

The hits only ever request default.html or index.html.

Are those the actual URLs requested, or the underlying filename? Do you even have URLs in default.html? Every last one of mine asked for / alone, except for that one possibly-human outlier, requesting a fairly popular interior page.

An intelligent tool * will figure out that if you merely want to check site accessibility or general response speed, asking for robots.txt will work as well as anything else, and will offend nobody.

* That sounds droll, but it can’t be helped.

[SumGuy]

If someone (or something) hit's my domain (http) without specifying or requesting a file, my http server (IIS4) will give a 301 to "https://my-domain.tld/index.html" (but it will log the request as being for default.html which is set as the "default" file which never gets served - but was once upon a time). The Abyss server operating on the https side will hand them index.html (and will log it as such).

If the hit comes in on my IP address (ie no domain is specified) then on http they get "The system cannot find the file specified." (and no 301 re-direction happens). I'm not sure what IIS logs in that case, but since I've tried it just now I'll look for it tomorrow.

If I try my IP address on https, my FF browser throws up "Warning: Potential Security Risk Ahead", the details being "Error code: SSL_ERROR_BAD_CERT_DOMAIN". My certificate is only valid for my domain, not my IP (and that's why the browser is concerned). Can a cert include an IP address as well as the domains? My cert specifies "example.com" as well as "www.example.com". If I tell the browser to "accept the risk" and continue, then Abyss server will apparently serve my site's landing page. Hmmm...

All that is interesting. I'm going to see if Abyss has a setting for serving (or not) a site when the request comes in for just the IP.

But for me that now raises the question -> how can I tell from the logs if a web hit was to my IP vs to my domain? I will look at the logs at the office tomorrow to find out.

Since you (lucy24) are blocking the Kaspersky Cerebro bot, it should raise a question as to if there are legit organic hits to your domain that are being "scared off" because their Kaspersky anti-virus software is telling them that your site is an unknown security risk. Or maybe Cerebro is silent on the matter. Kaspersky might operate a portal for anyone to check a domain's Cerebro reputation?

Cerebro does not request robots.txt because that does not accomplish its primary task - which is to check landing pages for malicious code. That's why it is impersonating an organic UA (although it is out of date and therefore of questionable value).

[lucy24]

how can I tell from the logs if a web hit was to my IP vs to my domain?

Do you log headers? If so, there will be--or won't be--a Host: header field. (I don't personally log headers on 301s, as it would be a different process and isn't worth the trouble for me, but it probably isn't very complicated.)

Seems to me that if someone is offering a bona fide useful legitimate service, they should provide some hints to the target site about who they are and why they should be allowed in. Otherwise, paradoxically, it's the sites that do let them in, no questions asked, that might be considered the most high-risk.

[Sgt_Kickaxe]

I like Kaspersky, they have a great root kit tool for finding the nasty Sinowal virus.

That being said I would still block Cerebro.