-- Search Engine Spider and User Agent Identification
---- possible botnet
lucy24 - 4:42 am on Jul 12, 2012 (gmt 0)
very unlikely and highly suspicious
There you go, talking about me again.
Another benign explanation: It's someone in a programming class, assigned to make a crawler. They're not interested in your content, so they just pick up robots.txt because the crawler has to bring back something to prove it's been there. For appearances' sake they might also pick up the front page. And then you never see them again. Over on an unrelated forum, a couple of people I know were talking about an online programming class that one of them was taking and the other was thinking about taking. The one who was thinking about it said, quote, I have no interest in making a web-crawler; I want to handle files and data.
Confirming a long-held suspicion that that's where some of them are coming from.
Incidentally, while looking up this post I found a discussion about a legitimate mail server that unfortunately lives right in the middle of a blocked range. The site administrator had to, in his words, "poke a hole" in the middle of the block so people could get their e-mail. I know the feeling.