Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?


wilderness - 7:20 pm on Oct 2, 2011 (gmt 0)


I think the /8 exceptions would work for an established site that wants to grant access for legitimate users overseas.


IMO a flawed and very bad decision, at least using that as your sole criteria.

For others; /8 suggests allowing by Class A.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com