Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Who is Kosmix?


blend27 - 1:47 am on Aug 18, 2010 (gmt 0)


websense: not just trying to parse every pos.. page and gets into bot trap, but also changes User Agent to random strings that most of the time don't make sence.

so I have a routine that checks:

if: IP contains 208.80.194.' or '208.80.195.' or '208.80.193.' or '208.80.192.'
then: serve status code 200, DISPLAY "Hello World", abort.

There might be more ranges, but that does it for me.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4188052.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com