Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?
tangor - 5:40 am on Oct 3, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
|people are likely to want to "crawl" the site by doing lots of search queries rather than link-based crawling. |
Separate issue... I have a few sites like this where I offer site-based search (no bots allowed, run my own index). Keeps it all "in-house".
Brought to you by WebmasterWorld: http://www.webmasterworld.com