Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?


inbound - 7:33 pm on Oct 2, 2011 (gmt 0)


Oops...

Told you I need help. I thought you meant allowing (first 3 * 8 bits of IP).anything(last 8) (which I mistakenly put as /8, it's a long time since I used such terminology and should have checked) - are you suggesting something different.

Do you mean opening up single IPs? Do you think that allowing class C blocks allows people to crawl from lots of separate IP's in that range (or are there other reasons why that's a really poor decision?).

Also, do you find that people change IP often in the States?


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com