Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
jmccormac - 3:01 pm on Oct 30, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4619880.htm
Bot detection now probably requires a good IP database along with some statistically based behavior rules. Otherwise these might go relatively unnoticed. How immediate is the JS risk in comparison to the existing scrapers?
Brought to you by WebmasterWorld: http://www.webmasterworld.com