Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- obot


lucy24 - 4:55 am on Jan 16, 2012 (gmt 0)


Well, dang and blast. Here I'd already gone and blocked them on robots.txt grounds, before ever consulting the horse's mouth [cobion.com] via their own links.

There we learn among other things:
IBM Proventia Web Filter database categories
<snip, snip>
Religion: Includes Web sites with religious content, information about the five main religions, and religious communities that have emerged out of these religions.
Sects: This category contains sites about sects, cults, occultism, Satanism etc.

Tough luck, all you Sikhs, Parsees and Baha'is. Guess you're just cults.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4376847.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com