UA: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) (Elisabot 1.0 unforgettable) Robots txt: no (or I didn't detect it) Host: xs.nl 80.246.176.0/20
incrediBILL
2:07 pm on May 5, 2014 (gmt 0)
Robots txt: no (or I didn't detect it)
It's quite common these days for the robots.txt probe to be a blank user agent and possibly a different IP due to cloud networks or shady crawlers using proxies.
Expand your search :)
lucy24
4:27 pm on May 5, 2014 (gmt 0)
Is "unforgettable" part of the UA string? I used to have an "And the winner is..." thread for names like that :)
### it, Elisabot, now I've got an earworm.
keyplyr
6:20 pm on May 5, 2014 (gmt 0)
Is "unforgettable" part of the UA string?
Yup... as posted.
keyplyr
8:28 am on May 15, 2014 (gmt 0)
It's quite common these days for the robots.txt probe to be a blank user agent and possibly a different IP due to cloud networks or shady crawlers using proxies.
Expand your search :)
Well, first I'm aware of that. Second, blank UAs get blocked. And lastly, I have a bit of code that pulls all requests for "whatever" and no request was detected from IP or zone related to this agent.