Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Spider Hunting or Site Building?


enigma1 - 8:21 am on Jun 12, 2009 (gmt 0)


I think, it's an ongoing effort so no I cannot say that for my part. There are also different areas bots target and the countermeasures in place can detect certain things far easier than others.

At the application level, scraping is getting harder to detect and block but at least hacking detection is a bit easier. Then with the server, you can never tell if there is something wrong with the host on time, that's perhaps another back door.

I am also getting false positives from time to time and so I always need to revise these countermeasures. Plus quite a few of these defenses are experimental and so I cannot just recommend them to others.

Hope you have a quick recovery Bill.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3931380.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com