Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- WebSense Hiding Behind Rotating User Agents


incrediBILL - 10:26 pm on Jul 10, 2008 (gmt 0)


They've only attempted to access 1700+ pages from my site this year which is reasonable for an internet security company in 6 months.

However, it's the deceptive way they attempt to do it which is why I've had them blocked for a long time with the rotative user agent.

The part that amuses me is they insist on using the "Konqueror" browser UA which is so obscure in the first place that it put a big neon sign out saying "LOOK AT ME! SOMETHING FISHY IS GOING ON! CHECK MY IP ADDRESS!"


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3692641.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com