Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?

tangor - 5:31 am on Oct 3, 2011 (gmt 0)


No matter what you do, if any link gets out, the bots will find it. And if found will crawl. Only way to deal with that is to ip ban (problematic) or password the product.

Securing the web is extremely difficult.

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com