joined:Sept 1, 2012
A new bot for me, from a company that Jim Cramer has been pumping the stock on his cable show.
22.214.171.124 - - [23/Jun/2014:20:42:45 -0400] "GET /robots.txt HTTP/1.0" 403 237 "-" "spyder/Nutch-2.1 (just another internet crawler; http://www.paloaltonetworks.com/products/features/url-filtering.html; email@example.com)"
I let all bots in through robots.txt, but ban them from going further with various .htaccess tests. This one failed for HTTP/1.0 and UA words spyder, crawler, and Nutch.
I have a suspicion of what the bot is up to regarding an obscure web site as my own, but no doubt experienced webmasters know precisely their goal based on their company services.
[edited by: incrediBILL at 4:31 pm (utc) on Jun 24, 2014]
[edit reason] formatting [/edit]