SteveWh - 5:51 am on Sep 19, 2011 (gmt 0)
I forgot a step in the outline above. The program also checks the URL against its list of website categories for parental filtering:
1.5) While the http request is still blocked, check the URL against the Parental Filter categories list. If the website belongs to a category blocked by the parental filter, put a warning message in the browser window instead of the requested page, and don't allow the http request to go out to that website.
It's when their only visit is anywhere from two minutes to an hour and a half after the human visit* that I'm scratching my head.
That could happen when the URL has been logged into the user's "URL History Log", a list that Trend doesn't receive until the next time the user's software auto-updates.
Wouldn't they have to live in your router to do all that pre-testing while being perfectly invisible in the logs?
It must have a means to detect and inspect the content of outgoing http requests before the operating system actually sends them out, which probably means installing a "hook" at the operating system level, and which many such AV programs probably do. It's the same sort of thing that a software firewall (as opposed to a hardware router) would need to be able to do.
Do you mean your router logs? I was using the Trend software firewall, which of course allows outgoing connections to the Trend server. If you have a hardware router that logs activity, that would certainly give info about how much (if any) of this behind-the-scenes traffic is really going on.
In your website log, you should see a request from the Trend bot and also a request from the person who's using a Trend AV product.
MSIE 6.0 is deprecated by MS so any serious use of it should, I think, be blocked.
IE6 is an old and vulnerable program. Malware embedded in websites often lies in wait for visitors who are using it, and attacks them (not attacking visitors who are using other browsers). When an AV scanner sends an MSIE 6.0 UA string, it is not using IE6. It is pretending to be a lame old weak browser. Provoking an attack by this method is a way to get shy/careful malware to reveal its presence -- when it attacks.
To be effective, viruses should be checked for on the "user's" computer at download time
As I said, it does that, too. The various AV companies have other strategies that they use in addition to that, in the attempt to gain a competitive advantage.
It's probably analogous to giving someone permission to open your mail
In years past, when the uploads of visited URLs first began being uploaded to the company server, there was no warning in the EULA about it, but nowadays there is.
If you block AV bots, it seems like that could prevent the AV companies from determining that your website is safe (but it seems very unlikely they would decide it's dangerous just because they're blocked), and it could prevent them from categorizing the site for parental filtering. That could potentially mean your site is sometimes blocked from access by users when, if the site were properly categorized, it wouldn't have been.