I've stopped blocking 38.0.0.0/8

[SumGuy]

I'm not sure who owned that /8 before 2016, but when hits started in April of 2016 they were clearly bots. Such as:

TurnitinBot (https //turnitin.com/robot/crawlerinfo.html)
HEADMasterSEO
Dispatch/0.11.3
Python-urllib/3.6
Mozilla/5.0 (compatible; evc-batch/2.0)

The black hole known as COGENT-174 seems to own the entire /8.

I first just gave them a 403, but by mid-2018 I had blocked that /8 in my router so I wasn't seeing them turn up in the web logs. There had only been 281 total http requests by then, so I figured this was still an evolving /8 that was largely un-utilized. But I still look at the router logs and run some of those IP's to see who's trying to hit me.

But lately I've seen attempts from 38.97.116.0/24 which is AS396527 MT-PUBWIFI Massachusetts Institute of Technology. I guess MIT doesn't have enough IP space...

AS396527 is announcing only 3 /24's, only 1 of which is in 38/8. What a way to dice up IP space. From what I can see, only 4.5% of 38/8 has been assigned. So I'm going to remove the block I have on 38/8 and see what else crawls in. Not looking forward to this...

[not2easy]

COGENT has always been a pitn about limiting info. Everything 38. is listed as "Cogent customers". Obviously some Cogent customers sublet their space to unwanted users. There is a mix of residential ISPs, businesses and government entities. Trial and error can help determine whet to block. It's the error part that is concerning.

[lucy24]

I just checked my htaccess, as I couldn't remember if I block the whole thing. Turns out I'm currently only denying one /18 and one /24 within that whole /8. Other sectors--including one right next to that denied /18--are to all appearances human. Still others are used by named robots FemtosearchBot (blocked) and MJ12bot.

Yah, trial and error indeed.

MIT could be absolutely anything. Not the link checker, though; that's still at 128.30.whatever-it-is.

[SumGuy]

Remember that I unblocked a /24 because it was assigned to an MIT wifi network. Well, I got a hit from that same /24 a few days ago.

38.97.116.244, 404, GET /.DS_Store, Go-http-client/1.1

Nice. Real nice.

I'll probably re-enable the 38/8 block.

[LifeinAsia]

I don't block the entire /8 range, but I pretty much have a 1-strike-and-you're-blocked policy for /24 ranges.

[not2easy]

That's what I do also. If I had an extra hour or so I'd put together a list of all of those 38.xs that I have had trouble with. One at hand is 38.242.226.0/24

Cogent's lookups add on a bunch of related info URLs, but they look like too much to wade through:
[rdap.arin.net...]
[rdap.arin.net...]

They also have some non-38.x ranges similarly blind so those are blocked too:
154.51.0.0 - 154.51.255.255 Direct Allocation
154.51.0.0/16
PSINet

I'd guess this bit of information barely scratches the surface.