G-i-g-a-b-o-t ignores robots.txt

may have been compromised

[keyplyr]

65.132.59.34 - - [08/Aug/2017:06:47:11 -0700] "GET /robots.txt HTTP/1.1" 200 7892 "-" "G-i-g-a-b-o-t"
65.132.59.34 - - [08/Aug/2017:06:47:11 -0700] "GET /example.html HTTP/1.1" 403 5049 "-" "G-i-g-a-b-o-t"

User-agent: G-i-g-a-b-o-t
Disallow: /

This happened dozens of times in a short frame, included in cluster of bot-net activity (all blocked) which would indicate G-i-g-a-b-o-t may have been compromised.

Previous discussion: [webmasterworld.com...]

[TorontoBoy]

I never saw that one before but I do have a Gigabot. You mean someone took over a banned bot? Is not that justice? What do you mean by compromized?

SetEnvIf User-Agent Gigabot keep_out

[keyplyr]

Gigabot = G-i-g-a-b-o-t, they changed the name a while back. This is a bot-for-hire. Anyone can run this bot, for any reason.
Host: gigablast.com
65.132.59.32 - 65.132.59.47
65.132.59.32/28
Parent: Quest ISP (CenturyLink Data Services centurylinkservices.net)
65.128.0.0 - 65.159.255.255
65.128.0.0/11
Note: This bot can also be downloaded from github and run from anywhere.

However, up until now (AFAIK) it supported robots.txt. And now it appears to be part of a bot-net. One hit from G-i-g-a-b-o-t, followed by 2 or 3 hits from spoofed browser UAs from various infected servers at other IPs amounting to approx 120 requests.

[lucy24]

Host: gigablast.com

Are they related to the bot that calls itself GigablastOpenSource/1.0 ? I found a few from
65.132.59.abc
in January 2016, but far more from a random array of other IPs both before and after. Haven't seen the Gigabot, with or without hyphens, in ages. (Poring through logs, I find a disproportionate number of requests for pages in roboted-out directories, so I shed no tears.)

[keyplyr]

Yes, all the same company, same range. Since their bots can be downloaded and run by anyone, you may still see older UAs in the wild. Theoretically, they may come from any IP.

Considering the events I reported above, these bot-net hits are likely *all* G-i-g-a-b-o-t using altered UAs on some hits to circumvent blocking.

The spoofed UAs were all from server farm IPs, so they were all blocked :)