Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- What's the Best Way to Keep All Spiders/Bots Out?


inbound - 6:30 pm on Oct 2, 2011 (gmt 0)


Thanks, I think the /8 exceptions would work for an established site that wants to grant access for legitimate users overseas.

There are a few problems that I can see with that for us though:

1. The site will be brand new so no previous publicity; I think webmasters will be interested in it no matter where they are from - so I don't want to alienate people at a time when they might want to look at it in order to discuss/write about it.

2. There might (still to decide) be tools on the site designed for webmasters, these should appeal to all English language site owners even though our (non-webmaster) audience would be very much UK specific.

3. I hope that traffic will be enough to make manual scanning of logs a bit of an issue.

I suppose that I really need to decide on whether the webmaster tools will be part of it - if so I could do an alpha and beta stage that could be invite only (with people getting a number of invites each); that could serve as a source for IP exceptions (that would still be subject to other anti-bot measures).


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4369764.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com