Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- WebSense Hiding Behind Rotating User Agents


Romeo - 12:18 pm on Jul 11, 2008 (gmt 0)


The part that amuses me is they insist on using the "Konqueror" browser UA which is so obscure in the first place that it put a big neon sign out saying "LOOK AT ME! SOMETHING FISHY IS GOING ON! CHECK MY IP ADDRESS!"

ahem, hmm, well ... why do you think that?

'Konqueror' is the default browser on a Linux KDE desktop, and since it is smaller and faster than Firefox, I am using it all the time for my private web surfing ...
Didn't know I am considered obscure due to this ... obscure, yes, but just due to this?

And in case the Konqueror may have difficulties in rendering some pages, I also have the Gnome 'Epiphany' browser on my Kubuntu desktop. Another obscure one, yes, I already know.

From the large variety of UAs mentioned in this thread, I would assume that they at Websense may just randomly pick them from their own web server log files to insert into their bot ...

Who are the users of Websense?
Would blocking their bot be like shooting in the own feet because our sites may get flagged and their customer base may be driven away?

Perhaps we should give them a blank 200 page or a 'Lore ipsum' page instead of a harsh 403.

Kind regards,
R.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3692641.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com