Mozilla/5.0 (compatible; Adsbot/3.1)

[tangor]

UA: Mozilla/5.0 (compatible; Adsbot/3.1)
Robots.txt: yes
IP: 216.18.204.140
Host: 216-18-204-140.hosted.static.webnx.com

Ignored robots.txt, grabbed 85 html in seconds, no images, no pdf, no css

Received a 403 as do any "ads" or "ads.txt" (or variations of same if they don't honor robots.txt

This particular site is ads free of any kind so even if it might be useful for others it has no value for me.

[lucy24]

They must be recent; I find a flurry of them starting just a few days ago, on the 22nd. Blocked due to deficient headers. Generally one or two requests per visit. Assorted interior pages, suggesting a referer from elsewhere (maybe someone who carries ads has a link to the page?).

I’ve just added a robots.txt disallow to see if they pay attention.

[wilderness]

There's a 2012 thread on the host.

[webmasterworld.com...]

[tangor]

Thanks for the eagle eye and sterling memory! I searched WW for the wrong query. Whew! (which resulted in Jakarta...)

[jmccormac]

Persistent maggot even after being blocked for tryng to scrape over 1K pages. Came from 6 IPs on webnx.com. (173.231.59.ddd).

Regards...jmcc

[CodeJockey]

Very persistent from last night to today. Spotted it last night in my logs. From webnx.com (173.231.59.214). I blocked it immediately which usually causes a loss of interest, but 10 hours later it's still gathering Forbidden. My guess it's done the better part of 40K off my site.

[lucy24]

I blocked it immediately which usually causes a loss of interest

I think it depends on whether they’re spidering from scratch or coming in with a shopping list. If they’re spidering from the root, and the root is blocked, then they really can’t get much further. But if they have a shopping list they will stubbornly ask for every last item on that list, even if it means collecting a library's worth of 403s.

[CodeJockey]

I watched the process eventually end, and thought that this was finished. No wait. "Whomever" must have realized what had happened and did a short burst of 'just in case' queries. (Just to verify that the 40K+ messages might be wrong.) So wait for a couple of hours and it looks like the process is happening all over again. Which confirms your comment that they've got a list. And they're collecting their 403s. Haven't seen this in quite some time.

[tangor]

Desperate days produce desperate ways ....

World wide lockdowns have provided more time and incentive for some bot wranglers. Sigh.

My bot activity IN GENERAL is up near 30% year on year.

Other aspect is I have just as much "free time" due to the global slow down so whack-a-mole has come back in vogue for afternoon pastimes. :)