Forum Moderators: open
... until today!
They came crawling using the following UA:
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)"
Now I'm installing hard blocks on them.
Here's the IP ranges involved in today's scraping:
67.202.31.* ec2-67-202-31-*.compute-1.amazonaws.com.
67.202.49.* ec2-67-202-49-*.compute-1.amazonaws.com.
67.202.50.* ec2-67-202-50-*.compute-1.amazonaws.com.
67.202.54.* ec2-67-202-54-*.compute-1.amazonaws.com.
67.202.57.* ec2-67-202-57-*.compute-1.amazonaws.com.
75.101.194.* ec2-75-101-194-*.compute-1.amazonaws.com.
More importantly, it was executing bot GET and POST which is notable because some of my navigation is in JS and uses a drop down form field just to thwart bot crawling.
Well, that ship has sailed as they appear to have executed that code a few times.
No, it wasn't human, the page accesses were too fast and high volume, about 600 pages total.