Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- British Library scraper
g1smd - 1:35 pm on Aug 16, 2012 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4484756.htm
I think I'd let that in for sites with factual content and that are going to be archived for long term storage.
Brought to you by WebmasterWorld: http://www.webmasterworld.com