Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?
inbound - 5:49 pm on Oct 2, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
I should add that hardware load balancers and firewalls are within the budget, so if any help in this respect (thinking of the ease of IP range banning) I'd be delighted to hear about your good experiences.
Brought to you by WebmasterWorld: http://www.webmasterworld.com