Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?
wilderness - 7:20 pm on Oct 2, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
|I think the /8 exceptions would work for an established site that wants to grant access for legitimate users overseas. |
IMO a flawed and very bad decision, at least using that as your sole criteria.
For others; /8 suggests allowing by Class A.
Brought to you by WebmasterWorld: http://www.webmasterworld.com