node-superagent/3.8.3

[hekasi]

"node-superagent/3.8.3"
52.214.211.116

My firewall has been blocking all the request from node-superagent/3.8.3. What is it? it looks for some random pages which are not in my website.

Thanks

[lammert]

Welcome to WebmasterWorld hekasi!

The node-superagent is a generic library to make HTTP calls from a node.js based server. It is not a bot with a specific purpose but a library which can be used for multiple purposes. I don't know of any respected search engine or on-line service using this library for legitimate purposes.

[hekasi]

Thank you mert,
This forum is such a helpful place.

So it is good to block it?

52.214.211.116 - - [24/Feb/2020:08:24:01 +0000] "GET /page/1/?s=%EC%97%94%EB%8B%A4%20%EC%8A%A4%ED%8B%B0%EB%B8%90%EC%8A%A4 HTTP/1.1" 403 31 "-" "node-superagent/3.8.3"

These are the kind of files this is looking for.

[edited by: hekasi at 12:21 pm (utc) on Feb 24, 2020]

[lammert]

Yes, I would personally block them. This seems like some sort of fishing expedition for vulnerabilities on your site.

[hekasi]

Okay. Thanks a lot

[tangor]

@hekasi... Welcome to Webmasterworld!

For some things you block first then research. You can change your mind if necessary. :)

[not2easy]

That IP is in the Amazon group of servers, so you can use the UA to block, which would block it no matter where it comes from or you can add the 52.192.0.0/11 CIDR to your unwanted list of IPs. Or both.

[hekasi]

Thank you @tangor. Yes that's what I do block unknown bots then research but for this one I could not find much information which is Why I created this topic.

@not2easy I am using stopbadbots.com plugin and it blocks all the request from node-superagent

Thank you

[lucy24]

Wow, you're meeting some elderly robots. I saw the title and thought Oh, yeah, I’ve met [and blocked] those hundreds of times ... but when I delved into logs, I actually haven’t seen "node-superagent" since 2017, trickling away in 2018. What I do see these days is "node-fetch" so if you're blocking by UA, you might just block "node-" and-that's-all.

[tangor]

@lucy24 ... I've been denying "fetch" for a zillion years (which catches the one under discussion. Is this a bad idea? "node-" would be more precise ... and would send me back to deal with the other UAs that have "fetch" in them.

[hekasi]

@lucy Yeah I have blocked node and it blocks all of them. My firewall also blocks "fetch" "crawl" idk if that is useful.

Thanks

[lucy24]

:: detour to current logs ::

Huh. The string “fetch” isn’t nearly as common as one might think. Maybe it's one of those name elements that has gone out of fashion. About 2/3 of them are the abovementioned "node-fetch" and most of the rest are Feedspot:

Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)

--which I vaguely thought I'd denied by name, but I don't find it, so all those 403s must be due to headers alone, yay. (I do track it under “Blocked Ebooks”, which I guess is why I recognized the name.) There's also a handful of

weborama-fetcher (+http://www.weborama.com)

(now there's a name to inspire confidence) and the odd

Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)

(it always amuses me when someone thinks they can get in by claiming kinship with something that has done nothing to inspire confidence in its own right). Over a year ago there was a lone

SD4M-fetcher (contact:[e-mail]; details:https://sd4m.net/)

which distinguished itself by being the only "fetch" that ever asked for robots.txt. This, in turn, makes it the only request in the "fetch" group that didn't elicit a flat 403. (The robots.txt, that is. The ensuing page request was blocked.)

Does FeedFetcher-Google even exist? All I find--at least in the readily accessible past year--is things claiming to resemble it.

:: cross-check of headers ::

In addition to certain header deficiencies, they tend to come from unsavory neighborhoods. This is known in linguistics as double markedness. Tralala.

[blend27]

@hekasi - Welcome to WebmasterWorld!

That UA does not have parentheses. Full stop.

@lucy24 - ...markedness. Tralala = !