homepage Welcome to WebmasterWorld Guest from 54.167.11.16
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Amazon robot?
Just noticed a whole lot of amazon robot activity
Baruch Menachem




msg:4433122
 2:31 am on Mar 25, 2012 (gmt 0)

Then I began googlilng about it and it seems to be a Very Bad Thing. My web page is kind of small and pathetic, mostly used for hobby and education. Why are Amazon Robots a Very Bad Thing? Do they suck serious bandwidth? Are they the domain of script kiddies out to do E\/il?

Anyway, I went so far as to ban the Amazon Robot domains.

Thanks for your help

 

Dijkgraaf




msg:4433766
 3:04 am on Mar 27, 2012 (gmt 0)

There are various bad bots coming from the Amazon AWS cloud, is it one of those? If so they don't actually belong to Amazon but just a client paying to host something in their cloud.

Yes, sucking up bandwidth without any benefit to the web-master of the site in question riles up some of the web-masters.
Some bots may also be stealing contents to serve up elsewhere or looking for copyrighted material.
Some try and masquerade as an ordinary user, don't identify themselves or their purpose or fail to obey robots.txt
The above behaviours will get them a label of a bad bot.

There are a few threads in the Search Engine Spider and User Agent Identification forum that talk about them.
For example Amazon AWS Hosts Bad Bots [webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved