lucy24 - 5:44 am on Sep 18, 2011 (gmt 0)

It seemed that the software would send a URL that I typed into my browser address bar back to Trend even before my browser could fetch the page. I used to see requests from the Trend IPs (150.70. is one set of them) showing up in my logs even before my own request (if I recall correctly), and definitely at other times just a second or two afterwards, even for files that were secret, only on my server for a few seconds, and that I only requested once and then deleted.

Some of it has to be attributed to the logging software. Sometimes my server seems to be confused about what exact second something happened, and will hiccup back and forth like a time machine overdue for maintenance.

I found an even more illuminating one on a different date. The human visitor reloaded the page after 10 seconds or so-- and these page loads must have been separately reported, because an hour or so later 150.70 also visited the page twice, each time handing over every last letter of the query string. The only difference is that the human request for piwik.php came with a referer, while the robotic followup didn't.

Further scrutiny* reveals that the isolated request for a stylesheet was preceded by the real page-- several hours earlier. This in turn came only a few seconds after the triggering human visit-- unusually soon for them. Did some uber-robot look over the trip logs, jump up and scream at an underling to go get that stylesheet? (If so, the underling may have been fired shortly afterward. There are actually two stylesheets associated with this file. They only picked up one.**)

But what kind of security are you providing when you don't even look at a page until more than an hour after the human visit that triggered your inspection? Anything can happen in an hour.


* The Regular Expression Is Your Friend.

** ... and that one was only in use for a few weeks. Which means, I suppose, that it will take up space in my htaccess forever so robots can be redirected to the consolidated version. The one 150.70 didn't get, then or at any other time. Sigh.