Forum Moderators: open

Message Too Old, No Replies

PrivateSearch

         

keyplyr

6:18 pm on Mar 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



UA: PrivateSearch/0.1.0 (Polite Nutch Crawler; grierforensics.com)
Protocol: HTTP/1.0
Robots.txt: Yes
Host: AWS
52.0.0.0 - 52.31.255.255
52.0.0.0/11

Web forensics research

tangor

6:40 pm on Mar 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Chuckles (on my side) as I've nuked "nutch" for many years. Not sure I'd change my mind for "polite nutch".

keyplyr

7:21 pm on Mar 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



IMO there's nothing inherently bad about Nutch; it's just a simple crawler anyone can freely use. It's what they use it for that I care about. I filter "nutch" and other generic UAs and let through the ones I like. So far I allow about 30.