NetcraftSurveyAgent from new Amazon EC2 range

didn't read robots.txt

[thetrasher]

IP: 174.129.134.nn
User-Agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [e-mail])

174.129.0.0/16 = AMAZON-EC2-5

[GaryK]

It paid one of my sites a visit last week. It hit the default root page an excessive number of times in one minute before it got the boot.

I'm getting ready to track down and ban the entire EC2 netrange. Would any of you guys happen to know that information already?

[GaryK]

Thanks, Don. ;)

[blend27]

[ws.arin.net...]

The ones that have the EC2 in the NETNAME

[incrediBILL]

Has anything other than nutch that originates from the Amazon range ever read robots.txt except to ignore it?

Just wondering out loud...

[GaryK]

blend, that's the same link Don sent me via StickyMail. :)

Bill, are you implying Nutch reads robots.txt?

[wilderness]

blend, that's the same link Don sent me via StickyMail.

Our site host and/or moderator get touchy about such broad links, thus, I thought a sticky the best protocol.

[incrediBILL]

Arg, you pranksters got me.