Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Simple user agent checks


tangor - 9:49 am on Jul 18, 2012 (gmt 0)


Your rules might be different than others... There are few hard and fast rules (other than banning known server farms by IP... and even then some might disagree).

You do for your site what is best for your site.

That said, any deny is a potential deny to a real live human being (potential! is! key! word!). Allow all, or allow only what you want (called white listing) where you only allow a specific set and reject the rest.

Whatever makes it easier to sleep at night is what you will do. Chasing bad actors and compiling a huge list is a lot of work... much easier to say who can come in the door than to say let everybody in Except this one, that one, that other one, that one other there, whoops, he's a friend of that one, and by golly there's another one on the same block at that first one... Black listing will makes you nuts.

Looking at your 1, 2, 3, 4 above, that's close to white listing. Investigate that concept. See if it fits with your intended audience...


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4476571.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com