Forum Moderators: open
[webmasterworld.com...]
...but sadly I couldn't add an extra follow-up!
Anyway had a quick look at the snapshot I'd taken of its data and wanted to add that the reason it seemed to be coming from the ISP (InCom.net) is that they were using their SQUID proxy server which masked their IP.
Their actual IP was 216.0.109.10 which resolves to "www.netnose.com" - this makes a fair amount of sense given the name of the crawler. The site gives noting useful apart from an authentication prompt.
All it did was grab one page without checking robots.txt
- Tony
