Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Updating bot ban list & cleaning out obsolete entries


wilderness - 11:59 pm on Dec 21, 2009 (gmt 0)


There is something in the Forum Library which Jan (aka bull) put a lot of work into and could be updated.

[webmasterworld.com...]

The problem with these things is a lack of conformity (as per of the old Close to Perfect threads), and especially when some dweeb comes along and inserts their entire version of an htaccess, and irregardless if the file is a functioning file or not.
The repetition and length of such submissions turns off most regular participants.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4046696.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com