Annoying Crawler, Bot, or what?

Malicious traffic from a certain ISP/network

[PeterTroyWilliams]

Hi Folks, Il try this again, and I hope I am posting this in the right place.
My site is being hit by Oculus Networks repeatedly , which gives our site a high bounce rate. They seem to be in the 172.252.***.*** range and a few others. They come to the site, and hit the same page about 30 times, leave and come back again doing the same thing. They all are Linux, Chrome 149.0, 1920x1080. It is annoying, as we use some free services at our site that here limited by hits, before they are not free anymore. I do use cloudflare , and have attempted to block them through cloudflare which dont seem to be working. Any suggestions would be greatly appreciated!
~Pete

[lucy24]

have attempted to block them through cloudflare which dont seem to be working

This is where we will need details. *Exactly* what have you tried, and what if anything is the result? (I don't personally speak cloudflare, but plenty of others do, so be specific.)

some free services at our site that are limited by hits, before they are not free anymore

Does it go by total requests, or only by successful requests? In some cases, blocking a request doesn't really save your server much work, unless the pages themselves are quite hefty. I'm assuming your robots, like most, are only requesting pages, no supporting files--or at least, no images.

<tangent>
In recent months, I too have been much vexed by robots requesting the same page over and over--generally clusters of requests from all different IPs and UAs, rather than multiple requests from the same source. Short of getting stricter about oldish browsers, and blocking the rare IP that is a server farm/colo rather than human ISP, there's not a ### thing I can do.
</tangent>

[PeterTroyWilliams]

Not sure if they are. The all have the same thing in common, Oculus Networks, which is a proxy I think,and the browser specifics. I have gone into cloudlflare and blocked individual ip adresses, (ip.src eq 40.223.131.239 and ip.src eq 172.121.149.193 and ip.src eq 40.223.203.229 and ip.src eq 40.223.213.113 and ip.src eq 40.223.212.140 and ip.src eq 172.252.237.166 and ip.src eq 40.223.127.209 and ip.src eq 40.223.219.108) but they just keep bouncing around the ips. so maybe be question would be how to configure a complete range?

[lucy24]

how to configure a complete range?

Well, that would depend on Cloudflare. Do they recognize IPv4 blocks in the form 162.121.128.0/19, or wider ranges in the form 172.252 ? (i.e. the same format you would use in your htaccess / config / IIS equivalent.)

Anything involving proxies or ISP ranges means you also risk blocking legitimate humans, so there's a tradeoff. If your robots are anything like mine, there won't be any obvious header deficits. Do you have the option of blocking by User-Agent? This too is risky, since some humans will be stuck in older browsers.

[not2easy]

Just picking out one of those - 40.223.0.0 - 40.223.255.255 = CIDR: 40.223.0.0/16 which was an Eli Lily range but WHOIS shows it as reallocated to IXPO and used as a proxy of Oculus Networks.

172.121.149.0 - 172.121.149.255 = CIDR: 172.121.149.0/24 (also IXPO and used as a proxy of Oculus Networks. )
Same thing with 172.252.237.0 - 172.252.237.255 = CIDR: 172.252.237.0/24

As mentioned, it does not help much blocking a single IP as the proxy can use another IP in the same range. You're stuck blocking every user of the proxy range or playing constant whack-a-mole. I don't use Cloudflare so I'm of no help there, but there are many CF users around who might have actual useful suggestions.

[PeterTroyWilliams]

Yes, they recognize they recognize IPv4 blocks in the form 162.121.128.0/19 It can be configured to pose a captcha rather than a block, making me really wonder now if by the capta choice, if it is still actually connecting to the site, making a "hit". Again, my only issue is the bounce rate , and the annoyance of statistics being all trown out of wack.

[PeterTroyWilliams]

Thank you. Yes these are the IP ranges of the offender. I will see if I can use Cloudflare to block the complete range.

[PeterTroyWilliams]

Thanks. I went into cloudflare and made a security rule to block those CIDR numbers and a few more, I also blocked the associated ASN. Ill know tomorrow . I learned what all that means today! thanx again.

[PeterTroyWilliams]

So nice! Im seeing normal hits again, instead of Oculus hitting the same page with different IPs all day. I have seen this before, and when your sites bounce rate goes up, seems google ranking goes down. I hope this fixes the issue. Thanks all, and especially to not2easy!

[Martin Potter]

Peter, I am seeing the same sort of visits to my website (stand-alone, not on Cloudflare). There must have been several hundred requests over a period of a couple of hours, asking for the same page over and over again (html only, none of the images there), from apparently random IP addresses, but including a small number in the 40.223.0.0/16 range that you noticed. Some of the requests got a "404" because I had already blocked that particular IP address range, but many others were fulfilled (200), for what purpose I know not. All of this action has just about doubled my normal bandwidth usage for the today!

[lucy24]

All of this action has just about doubled my normal bandwidth usage for the today!

As a rough-and-dirty metric, I can estimate traffic by the overall size of an Apache log file. For my main site it used to run about 1MB daily; lately it’s been 2 or 3, up to 4. And about half the total log size used to be 403-or-similar (418, 429, 400 ... gosh, what a lot of ways to send a bad request); lately it's more like 2/3 or 3/4. Which does mean I'm succeeding in blocking a decent proportion of them, but how I wish they'd go away.

On the other hand, the number of requests is nowhere near so vast that I'd suspect a DDoS attack--and in any case, why me? (Or, I guess, why my host's server?)

[PeterTroyWilliams]

@ Martin Potter
I Thought cloudflares fix did it...but it didnt. I added the following code to my .htaccess file and it now blocks those ranges...
#BEGIN Blacklist
<RequireAll>
Require all granted
Require not ip 40.223.0.0/16
Require not ip 172.121.0.0/16
Require not ip 172.252.0.0/16
</RequireAll>
# END Blacklist

[PeterTroyWilliams]

I think it is someone from a free plugin that offers so many free visits and then wants to charge you for a premium. What it seems to be is someone using a meta refresh on an Iframe containing a page on my site, and then going through a proxy to hide their location.

[not2easy]

I don't know if the format is the same as it was for the older Apache version (your blacklist is in the Apache 2.4 format) but there should be an equivalent available for blocking cross-site XFrame efforts. Something lucy24 would know, something for

Header append X-FRAME-OPTIONS "deny"

to prevent others from using Iframe to show your content.
_{Sorry, I'm on my old laptop today and it would take too long to find newer references.}

[lucy24]

Nope, not a header I'm familiar with, but it can't hurt. (And I'm pretty sure directives involving Header haven't changed between 2.2 and 2.4.)

:: brief side trip to Apache docs, followed by longer side trip to developer.mozilla.org ::

Possibly we want "SAMEORIGIN" rather than "DENY", on the off chance that something on the site uses internal framing. Disclaimer: This header wouldn't prevent a visitor from downloading the material; it would only prevent them from viewing it. And I’m not sure that makes a difference when there's a robot behind it all instead of a human browser.

:: business with htaccess to confirm that neither value of this header affects display of analytics opt-out box, which lives on a different site ::

[shawnb61]

Oculus (AS398781) is similar to Oxylabs (AS396319)... Their business model is to provide bots with services to make them hard to detect.

This is why blocking single CIDRs doesn't work. These guys BRAG about the fact that they spread their clients' activity around, to make bad activity harder to detect & block. They do it automatically. It's the very service they provide. Heck, if you pay more, they can make you look like you're a normal US residential IP address.

Lookup their FAQ, on their website... On their pricing page... A few direct quotes:

Why do I need proxies:
Proxies are a necessity for any bot or tool when scraping or trying to purchase something from a website. Proxies allow you to run more tasks, decrease the odds of being banned, and have a faster connection to most websites.

What is an ISP Proxy?
This is the highest quality Residential proxies leased from Internet Service Providers (ISP) in various countries that you can purchase. This plan works great on eCommerce websites with bot protection such as Shoppy, Walmart, Target, Best Buy, Amazon, and more.

What is a ISP Premium Proxy?
This is the highest quality Residential proxies leased from US Internet Service Providers (ISP). This plan works great on Google related and key social media websites allowing for unblocked access from US Residential IP addresses.

What is an Datacenter Proxy?
Oculus Data Center Proxies is a shared DC plan which comes at a very affordable cost. This product's main purpose is for web scraping.

[lucy24]

:: wanders off sobbing brokenly ::

[shawnb61]

They even offer special services for automating event ticket purchases & sneaker purchases.

[shawnb61]

I am in 'block all datacenters' mode. I help out on a small site, but man do they want our data.

I have no problem with humans visiting the site, of course. And search engines that actually generate traffic.

For all its faults, our traffic appears to be mainly generated from Google and links from other sites. (Search engine breakdown is 81% Google, 5% Bing, 14% misc.) We somehow still get pretty good ranking on Google, I think because we're kinda niche. We even show up *above* sponsored results on certain queries, mainly, for queries regarding bugs/issues/failures/problems, which the sponsors don't want to be associated with. And that's one of the main things folks come to us for, the stuff the vendors won't fess up to...

In an ideal world, we would find a simple rule to block everything that isn't - truly - a residential IP. Allow Google/Bing/Duck thru. And that's it.

I see no reason to allow any traffic from any datacenter that supports commercial-only customers.

Today, I block by ASN. The bad behavior is at the org level. Blocking by CIDRs is a waste of time. You have to block ALL the CIDRs for orgs that provide services and safe haven to these bots.

My host uses mindmax, and recently exposed ASNs as server vars that I can use in .htaccess. Folks like Oculus & Oxylabs take ONE line each to block now.

Before my host provided this, I used the freebie DBIP datasets, and generated a list of all CIDRS for my list of bad ASNs. I wrote a utility that combines and cleans the list of CIDRS, eliminating overlaps & consecutive ranges, and produces the cleanest, shortest list of CIDRs mathematically possible, ipv4 & ipv6.

It'd be much nicer to have a residential-only whitelist, though... More thorough, less whack-a-mole... That's my grail quest, I guess...

In theory... My understanding is that hosts are REQUIRED to segregate residential-only, and report this to ARIN. In that simple (likely naive) reading of the rules, that data is out there. I believe in the old days this was to make it easier to identify email spammers.

And... In theory... My understanding is that this is what the Spamhaus PBL is - residential-only IPs; the list is generated to make it easier to block suspicious email sources.

I'm going to try to get my hands on the PBL... Do some analysis & see if it might help... It'd be good to know if it takes a bite out of the problem.

Because, even blocking all of Oculus, they still offer that premium service where they're using other ISPs' IPs...

[SumGuy]

How are you blocking residential proxies?

[SumGuy]

BTW, Eli Lilly owned 40.223.0.0/16 starting in 1991 but in May last year the entire /16 was transfered to - ta da - IPXO.

You can boil down all the CIDR's listed for Occulus (AS398781) to 377, of which 114 I wasn't previously blocking. I wasn't blocking the entire 40.233 but I am now.

I was starting to see a lot of hits from Occulus / 40.233 at the end of June. They all seemed to have this UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

Which was already in my bot-detection rules.

Occulus AS398781 has a single peer: Something called "Hosted Backbone" AS14713. That AS has 190k IP's spread across 227 CIDR's of which only 3 I was not blocking.

[shawnb61]

How are you blocking residential proxies?

I don't. That's my grail quest.

It sure feels like that info is out there...

What we have done, all via .htaccess:
- Blocked all of China... Didn't want to do it, we had active users there... But it was impossible to support that volume on our hosting plan, & it just felt like a new challenge every day.
- Blocked ~30 problematic high-volume ASNs, after confirming either (a) we had no registered users there, or (b) that org doesn't offer residential service.
- Blocked all of Microsoft & Google, while allowing GSC, googlebot & bingbot & duckduckgo. GSC works fine & our search rankings are still good across Google, Bing & Duck. (bottom of [webmasterworld.com ])
- Blocked any & all wp file requests... We're not a wp site...
- Blocked any & all .env requests...
- Blocked any & all .php requests that aren't properly formed app requests.

Once a month, we recast our blocked ASN list into a new/current set of CIDRs for that set of ASNs. Those IPs change hands all the time. Not too surprisingly, they seem to shuffle back & forth among the bad players.

I've also automated disabling certain app functions when any ASN exceeds a very high volume threshold within a 10 min period: disabling php session management & pretty much all app-side logging/stats. If an unblocked ASN hits us with crazy volumes, they're not disabled, however, there are zero DB writes, just reads, and impact is negligible. We just lose some stats, e.g., their page view counts aren't captured. Meaningless anyway. This action gave our CPU charts a buzz cut.

We do not block by individual CIDR or IP anymore. The org is the culprit; block the org. They either are a bad guy, or they knowingly harbor bad guys & don't police their customers (as long as they pay their bills). No playing whack-a-mole.

The biggest gap, of course, remains the attacks from what appear to be residential IPs. The online equivalent to using civilians as shields.

We regularly get high crawl volumes from orgs where we have a lot of current, active registered users, e.g., Contabo, M247, KPN, Cox, Datacamp. Can't block those by ASN. I'd sure love to find a way to automate blocking that particular crawl activity without impacting users.

I had held out some hope for the Spamhaus PBL, but you can't download that...

[shawnb61]

I was starting to see a lot of hits from Occulus / 40.233 at the end of June. They all seemed to have this UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

I think that's a valid mac UA. Very commonly used, by actual people & bots trying to look like people.

Note also that Oculus doesn't write bots. They offer bot-friendly hosting & network services. So...

You cannot rely on a UA to identify them. A different bot author (/Oculus customer) will do something different... Over time, you will see different behaviors from AS398781.

[lucy24]

You cannot rely on a UA to identify them.

On the other hand, the older a humanoid UA, the likelier it is to be a robot. When did you last meet a human using MSIE 6? But it's very much a ymmv situation, since some sites by their nature have an unusual number of users with elderly equipment (reservation schools is one textbook example).

Blocked all of China... Didn't want to do it, we had active users there... But it was impossible to support that volume on our hosting plan, & it just felt like a new challenge every day.

Happily, most Chinese robots don't think to change their language header, so that knocks out most of them before having to resort to IP. (And, doubly happily, most of China's CIDR ranges are exceedingly wide.)

[SumGuy]

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

The first time that UA shows up in my logs is Nov 13 last year. I only see it a handful of times up until Dec 9, but they're all legit hits.

Then on Dec 16 I see a hit from 40.223 using that UA. It's doing wierd stuff like GET / HEAD pairs.

The next few days I see that UA paired with Accept-Language zh-CN,zh;q=0.9 asking for my pdf files (referrer is scholar.google.com). That's when I probably started blocking language = zh-CN,zh;q=0.9 by Dec. 20.

Looking at what's happened since then, I'm convinced that that UA fingerprints as bot 100%. Go and check for it in your logs. Find an example, run the IP at spur.us and see if they confirm residential proxy.

In looking at these hits just now (using that UA) sorting the IP's, I've just spotted 69.91.188.0/24 and 69.91.190.0/24 as standing out. An example - 69.91.188.204. It's a Comcast IP, but Comcast gives 69.91.188.0/24 it's own announcement. I run any of those IP's at spur - they come up as Oxylabs. So those /24's are now in my IP block list.

And look what else turned up - a few hits from 69.12.58.0/23 -> "Twitter" !

LOTS of my rules are based simply on seeing "Chrome/x" in the UA where I have maybe 2 dozen different values for x.

I admit it has been a battle blocking these residential VPN's based on request-header rules but I haven't tinkered with my rules for probably 3 months now and they're catching every bot using these residential proxies.

[lucy24]

That's when I probably started blocking language = zh-CN,zh;q=0.9

Heh. I just block ^zh, and then poke a hole for ^zh-TW.

[shawnb61]

This exact UA, byte-for-byte:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

...Is in my logs the last few weeks ~532K times, across most ASNs. I have thousands of rows of valid, current, registered, end-user activity using that UA on my site.

It's a common, macos UA, I think for Catalina. Not current, but not too old either. You can verify by just doing a search for it.

But... That's exactly why bots love it. They are disguised as someone using a mac. ~45% of mac users are still on Catalina (according to a quick search, I'm not a mac person...)

Only a tiny percentage of the 532K requests is from Oculus IPs. I see this activity from Cox, GTT, M247, Bell Labs, Host Royale, Datacamp, Verizon, etc. The usual suspects. The ones I can't block because I have far too many users there.

I'm sure some of it is real traffic, but to me it looks like at least 95% of that activity is crawl activity, methodically going thru all topics on my site.

(I load my web access logs into a DB for exactly this type of query & cross checking... I assign every request with an ASN & country via the DBIP freebies for analysis.)

spur.us looks interesting, but they reject my emails as non-corporate... Which is odd... I'm not allowed to register for a free account.

[SumGuy]

I just had a brief conversation with a bot regarding the user-agent in question:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

According to what I was told, "On macOS 10.15.7 (Catalina, Intel) you won’t get newer Chrome builds like Chrome 142—Chrome stopped supporting Catalina starting with Chrome 129 (requires macOS 11+)"

I then asked: Are you saying that seeing "Intel Mac OS X 10_15_7" and "Chrome/142" in a user-agent is unlikely or implausible? The answer:

Yes—it’s unlikely/improbable in a practical sense.

MacOS 10_15_7 (Catalina) is too old for Chrome 142 updates to have been supported on that platform. Chrome 142 implies the browser was able to stay updated to that version on that OS, but Chrome stopped supporting Catalina earlier than that.

So if you see a UA containing Intel Mac OS X 10_15_7 and Chrome/142.0.0.0, the most likely explanations are:

1) the UA string is being spoofed/fabricated (by the site, a bot framework, an extension, or a proxy/tool)

2) or the UA string is a mixed/incorrect logging value (e.g., part from one source, part from another)

3) or the device is not actually on Catalina in reality (or the OS token is wrong), even if the UA says it is.

I asked further if it was possible to conjecture if an Intel-based Mac was older or younger than a given year whether it could be upgraded beyond MacOS 10_15_7 (and hence would be able to run a more recent version of Chrome higher than 128) but could not get that answer.

But to me, this at least raises the possibility that the UA in question is a forgery.

[lucy24]

fwiw, my Intel desktop maxed out at Sequoia (13.7.8), which comes through in UA inexplicably as 10.15 (decimal point, not lowline). Safari is up to 18 with this OS, so no reason to allow any lower number.

Also worth noting is that for quite a while now, Firefox identifies itself as number.0 (150.0, 151,0 and so on), ignoring any sub-versions. I think Chrome generally does the same, but I haven't paid as close attention.

Curious that so many robots claim to be Mac, when the Mac has traditionally been less prone to infection of any kind. Do they think that this alone lends plausibility?

[shawnb61]

I have ~15 active end users who have used that UA over the last few weeks. Our site attracts a lot of EE types, tinkers who do things like build their own guitar FX pedals. Those guys are predominantly mac, and tinkers like browsers like Vivaldi. Vivaldi UAs are very similar & are now up to Chrome 150. I see Brave UAs are nearly identical & at Chrome 143.

The IPs for these 15 users are mainly thru AT&T & Bell Canada, some others. Activity is real user activity.

As for all that crawl activity that uses that UA, you're probably right, it may have been cobbled together, or randomly chosen. Checking up on those IPs raises red flags.

But blocking that UA would definitely block real users.

The challenge is that users are all over the map... Humans are messy, & not everyone can afford to keep their gear current... Before blocking old browsers a while back, I did some analysis of what my users were using, and... One of my regulars uses an old PC with a Chrome/80 UA...

I just checked, and, years later, that crazed old metalhead still has that odd useragent...
Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36

(He's normally very helpful, except for those days it appears he's fallen off his meds...)

[edited by: shawnb61 at 5:38 pm (utc) on Jul 22, 2026]