We continuously scan the internet for security research purposes. Our crawling is not malicious and only notes summary information for a page
source: http://ltx71.com
lucy24
4:00 am on Apr 12, 2018 (gmt 0)
They’re on my “barred anyway” list, meaning that they ask for robots.txt on every visit ... and proceed to ignore it on every visit. I don’t think “security research purposes” merits an exception.
