Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?
tangor - 5:31 am on Oct 3, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
No matter what you do, if any link gets out, the bots will find it. And if found will crawl. Only way to deal with that is to ip ban (problematic) or password the product.
Securing the web is extremely difficult.
Brought to you by WebmasterWorld: http://www.webmasterworld.com