Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
keyplyr - 3:23 am on Jul 30, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4597558.htm
The reason I've always blocked them is because they scrape content without permission. The fact that they (after the fact) then cite the source is no consolation. I feel it dilutes the uniqueness of my site.
Brought to you by WebmasterWorld: http://www.webmasterworld.com