Forum Moderators: open

Message Too Old, No Replies

Zauba Crawler

         

lucy24

6:53 pm on Mar 29, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hot off the presses:

IP: distributed across the usual suspects
robots.txt: no
UA:
Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io)
The URL in the UA redirects to a godaddy parked page, which ranks up there in confidence-inspiration right alongside the .io (British Indian Ocean Territory) domain. That makes four red flags mentioned in this forum in the past week. I didn't bother trying the email.

Without going into detail: This robot sends sufficiently humanoid headers that its requests (for interior pages) were not blocked. Fortunately it's got a name, leaving the bad_agent fallback, mwa ha ha.

keyplyr

7:39 pm on Mar 29, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Please post range(s)

My server doesn't respond to "usual suspects"

lucy24

7:55 pm on Mar 29, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



So far today, with nice shiny new 403s after I popped in the bad_agent variable:

34.229.126.abc
34.223.135.abc
52.91.185.abc
54.162.195.abc
204.236.204.abc

In each case, the same final abc, so exactly five IPs in total. All AWS, I think. Each IP is used many times; they don't use one for a few minutes and then move on.

I checked the headers. Everything there that should be; nothing there that shouldn’t.

Like I said: the usual suspects.

keyplyr

8:14 pm on Mar 29, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



UA: Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io)
Protocol: ?
Robots.txt: No
Host: AWS
34.192.0.0 - 34.255.255.255
34.192.0.0/10
52.84.0.0 - 52.95.255.255
52.84.0.0/14, 52.88.0.0/13
54.160.0.0 - 54.175.255.255
54.160.0.0/12
204.236.128.0 - 204.236.255.255
204.236.128.0/17

lucy24

8:50 pm on Mar 29, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Protocol: ?
HTTP/1.1

Hostname used: example.com (lower case, without www: correct for this site). In some cases, this detail may be relevant.