Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Data reseller bot shortlist?


keyplyr - 4:22 am on Dec 11, 2012 (gmt 0)


Most of us block unfavorable companies by IP range along with a UA list of tools (downloaders, email address scrapers, etc) that anyone can use.

For a true whilte list, you have a list of allowed browsers and bots (verified IP ranges and/or header info) then block everything else. This takes a higher level of programing skill.

Difficult to post a list since not all are unfavorable to everyone.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4526703.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com