Msg#: 3742499 posted 8:51 am on Sep 11, 2008 (gmt 0)
No clue, but I boot anything claiming to be a browser with "HTTP:" in the user agent, and the "User-Agent:" thing as well. I whitelist the top 4 SEs first since they have HTTP paths in their user agents and then anything else with HTTP goes into the trash.
Msg#: 3742499 posted 1:44 pm on Oct 21, 2008 (gmt 0)
Based on the behaviour I've seen from the UA starting with "User-agent", many of the sessions look like the AVG LinkScanner security pre-fetching we discussed recently.
Because of this, I've been serving a small page with a note about our site not supporting pre-fetching. The page includes CSS and image references which are never fetched by these user-agents. However, there is frequently another session immediately following from the same IP address, where the user-agent and browsing behaviour is completely normal and human-like.
However, after taking a look at Tapestry, I've modified the logic to block any UA starting with "User-agent" unless it starts with "User-agent: Mozilla/4.0 (compatible; MSIE 6.0;". This would block the UA being discussed here based on the MSIE version -- without requiring a "T5"-specific pattern, although it might be a good idea to include such a pattern.