Multiple hits, servers, UAs:
hubjoba01.asbnva1.hubpages.com [
projecthoneypot.org...]
hubpages.com
robots.txt? NO
svr101b06.asbnva1.hubpages.com
Apache-HttpClient/4.1.1 (java 1.5)
robots.txt? Yes BUT immediately ignored full Disallow. E.g., same day:
10/0n 11:19:02 /robots.txt 200
10/0n 11:19:03 /dir/filename.html 403
10/0n 14:16:33 /robots.txt 200
10/0n 14:16:33 /dir/filename.html 403