- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- British Library scraper
tangor - 9:01 am on Aug 16, 2012 (gmt 0)
Another example of why whitelisting is the way to go. Pick your battles, and manage them before the "war" starts.
Thread source: http://www.webmasterworld.com/search_engine_spiders/4484756.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com