Msg#: 4513742 posted 12:11 am on Oct 31, 2012 (gmt 0)
:: detour to raw logs ::
Did yours use auto-referers throughout, as in your examples? Mine did. Same IP you met, couple of weeks back. I've got the group as 18.104.22.168/20.
They only asked for the top level of directory files-- that is www.example.com/ and then www.example.com/directoryname/ --but they must be talking to some other robot, because there was one interesting 404. (Only meaningful if you know the exact site configuration and filenames.) This detail always worries me.
I also turned up an apparent human from 67.217.32.nnn a couple of weeks before that one. By "apparent" I mean that they scooped up a single page with all associated files in a reasonable time period. In retrospect they were slower than you would expect for a human: ten tiny images totaling 66K shouldn't span two seconds. I don't think a lot of people run robots on satellite internet. (Caching from sites in northern Canada doesn't count.)
:: further detour here to check whether any legitimate UA contains no spaces at all (answer: yes) ::