Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?


Pfui - 10:48 pm on Oct 3, 2011 (gmt 0)


You might have already considered this but here goes:

Put the static pages and common files in / with relatively loose root-level .htaccess controls, then put the search interface (and any supporting files) in a subdir with tight dir-level .htaccess controls? Or alternatively, in a subdomain?

I do the latter with a search interface and its databases and symlink the subdomain's .htaccess to the main site's because I want common blocks. (The files could just as easily be distinct but that would require much more maintenance.) The subdomain search interface is public but the front-end 'form' requires the same-domain referrer to work. FWIW


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com