Previously discussed here:
[
webmasterworld.com...]
(We will not talk about how long it took me to work out that the reason the thread is locked is that July 2020 is not earlier this year, less than six months ago. Oops. I seem to have misplaced a year.)
IP: 173.231.60.xyz
UA: Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)
I'm not 100% sure this is the same robot; note the slightly expanded UA string. Also slightly different behavior w/r/t robots.txt. I tested this one, found it compliant, and let it in. Since then it has been racking up 403s ... because it insists on sending a root / referer http://example.com/ for all requests except robots.txt and the root itself (the latter redirected from http to https).
Nope, I have no intention of poking further holes. If you want to look at interior pages, don’t lie about who sent you.
Is it possible they have got Referer: and Origin: mixed up? Still no good, since the Origin: header is pretty meaningless if you're a robot. Besides,
cursory research [developer.mozilla.org] suggests you're not supposed to use Origin: on same-site GET requests.
