iamzippy - 6:15 pm on Mar 10, 2012 (gmt 0)
Excellent post. Just about everything in there chimes with me. Especially Websense.
Websense is a self-appointed arbiter of what's good or bad on the Web. It has a customer base from whom it takes real money in exchange for access to its classified ratings database. Its droids obey a common set of rules:
Never ask for robots.txt
Always ask for the home page, and never send a referrer
Never fetch any assets linked-to from the page
Always specify an MSIE UA, of 7.0 or earlier (right down to 1.x)
Never use the same UA twice in succession
Follow 301 permanent redirects but never remember the new location (and switch UAs on-the-fly)
15% or more of the time, let it be known the client is festooned with adware or malware (ZangoToolbar, FunWebProducts, Hotbar)
Websense! Die! Die! Die!
And the auto-referrers? Well they're hoping the 'referring' URL is gonna show up somewhere, maybe in your analytics?
Auto-referrers! Die! Die! Die!
Tip o'hat to you for that one.