Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
dstiles - 8:54 pm on Oct 18, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4375642.htm
I'm seeing the occasional one. I have the system set to return a 403 but not block the IP. Seems a reasonable compromise.
But the concensus is correct: it's time someone came clean on this one.
Brought to you by WebmasterWorld: http://www.webmasterworld.com