Zauba Crawler

[lucy24]

Hot off the presses:

IP: distributed across the usual suspects
robots.txt: no
UA:

Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; [e-mail])

The URL in the UA redirects to a godaddy parked page, which ranks up there in confidence-inspiration right alongside the .io (British Indian Ocean Territory) domain. That makes four red flags mentioned in this forum in the past week. I didn't bother trying the email.

Without going into detail: This robot sends sufficiently humanoid headers that its requests (for interior pages) were not blocked. Fortunately it's got a name, leaving the bad_agent fallback, mwa ha ha.

[keyplyr]

Please post range(s)

My server doesn't respond to "usual suspects"

[lucy24]

So far today, with nice shiny new 403s after I popped in the bad_agent variable:

34.229.126.abc
34.223.135.abc
52.91.185.abc
54.162.195.abc
204.236.204.abc

In each case, the same final abc, so exactly five IPs in total. All AWS, I think. Each IP is used many times; they don't use one for a few minutes and then move on.

I checked the headers. Everything there that should be; nothing there that shouldn’t.

Like I said: the usual suspects.

[keyplyr]

UA: Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; [e-mail])
Protocol: ?
Robots.txt: No
Host: AWS
34.192.0.0 - 34.255.255.255
34.192.0.0/10
52.84.0.0 - 52.95.255.255
52.84.0.0/14, 52.88.0.0/13
54.160.0.0 - 54.175.255.255
54.160.0.0/12
204.236.128.0 - 204.236.255.255
204.236.128.0/17

[lucy24]

Protocol: ?

HTTP/1.1

Hostname used: example.com (lower case, without www: correct for this site). In some cases, this detail may be relevant.