Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Does amazon have a crawler or are all Amazon AWS IPs fair game?
Bewenched - 9:35 pm on Nov 5, 2012 (gmt 0)
I've busted several amazon "bots" scraping our site for images.... makes me wonder if they are actually stealing product images from sites for their own use.
Thread source:: http://www.webmasterworld.com/search_engine_spiders/4514970.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com