Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Stale bad bot lists


incrediBILL - 6:00 am on Mar 25, 2012 (gmt 0)


The theory is to DENY ALL, and than make exceptions for the visitors you choose.


Exactly.

Deny All.

Then punch a hole in the 'firewall' for browsers and allowed bots only. Cell phones are the only thing that's kind of tricky, and using browsecap.ini can help with that.

Everything else bounces off the website like it was a grade A circus trampoline.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4432835.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com