-- Search Engine Spider and User Agent Identification
wilderness - 11:04 pm on Aug 2, 2013 (gmt 0)
Are they over the top or malicious in roboting? One can spend a lifetime blacklisting when whitelisting is easier to control.
If this debate is going to be introduced into each and every thread, than this forum should definitely be shut down. When the forum topics are focused primarily upon whitelisting, there are simply not any threads introduced, and the few that are introduced are not discussed because the syntax for whitelisting is not available.
It's bad enough this forum remains moderated, and each new thread must be pre-approved, with a policy that was instituted back in 2003 or 2004 due to participants that are no longer even participating in this forum.