Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Does amazon have a crawler or are all Amazon AWS IPs fair game?


bigtoga - 9:53 am on Nov 2, 2012 (gmt 0)


There is no valid reason that someone should hit my site from an amazon AWS box. I'd love to just block the whole of amazon AWS/etc straight at the firewall. But I'm worried that I'll somehow block an amazon crawler/spider and that would possibly impact the sales I do on amazon.

Anyone have any suggestions/links for this sort of thing? I want to allow, if it exists, the actual amazon company to browse the site but block amazon's AWS/etc customers who spin up a server then scrape/spam with it.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4514970.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com