Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- 150.70. revisited


lucy24 - 6:47 am on Sep 19, 2011 (gmt 0)


In your website log, you should see a request from the Trend bot and also a request from the person who's using a Trend AV product.

That was my point. First comes the human. Then, anywhere from a few seconds to an hour and a half later, comes Trendmicro. With other nastiness-checkers I see them in tandem.

And I still haven't figured out the favicon. Why do they need two of them? Why do they need it at all? Surely a quick look at the HEAD would tell them that it hasn't changed since yesterday, and that it's the canonical 318 bytes rather than something suspiciously larger.

:: vague mental picture of entrance to robot headquarters lined with matched pairs of favicons ::


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4363886.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com