Msg#: 4433120 posted 2:31 am on Mar 25, 2012 (gmt 0)
Then I began googlilng about it and it seems to be a Very Bad Thing. My web page is kind of small and pathetic, mostly used for hobby and education. Why are Amazon Robots a Very Bad Thing? Do they suck serious bandwidth? Are they the domain of script kiddies out to do E\/il?
Anyway, I went so far as to ban the Amazon Robot domains.
Msg#: 4433120 posted 3:04 am on Mar 27, 2012 (gmt 0)
There are various bad bots coming from the Amazon AWS cloud, is it one of those? If so they don't actually belong to Amazon but just a client paying to host something in their cloud.
Yes, sucking up bandwidth without any benefit to the web-master of the site in question riles up some of the web-masters. Some bots may also be stealing contents to serve up elsewhere or looking for copyrighted material. Some try and masquerade as an ordinary user, don't identify themselves or their purpose or fail to obey robots.txt The above behaviours will get them a label of a bad bot.
There are a few threads in the Search Engine Spider and User Agent Identification forum that talk about them. For example Amazon AWS Hosts Bad Bots [webmasterworld.com...]