Spider running as Trojan?

Unintentional crawling without knowing it?

[Yidaki]

Might sound stupid, presumably it is. I had a bunch of visits from a german dsl line yesterday night, requesting more than ~30.000 pages within ten hours from 5 of my domains (various niche directories and online databases as well as a forum - all from different servers). The hits increased by the morning to more than 10 requests per second until i noticed it and blocked the ip. Ignored all robots.txt (in fact never requested one) and produced a lot of 404 errors because of bad url parsing of the "spider".

In the beginning of the attack the dumbhead submitted one of his own sites to one of the directories, leaving his contact information (imagine that!). I tried to phone him at his official business phone but he didn't pick up the whole day long. Today i tried again several times and he finally called me back.

He acknowledged that he submitted his site but denied that he ran any spider. I'm convinced that he's lying.

His User-Agent has been Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt) for the crawl and Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.11 [de] when he submitted his page plus some more clicks simultaneous to the crawl. I'm pretty sure, the first agent is spoofed. All hits - including the crawl hits - were using anonymized referers (these strings like KVFVWRKJVZCMHVI... produced by Norton).

I informed the isp to take appropriate actions. However, before going any further with him i just want to ask you fellow trackers, if there's any known trojan or any virus on windoze that crawls pages you visit and external pages that are linked from there as well? Is it possible that someone has such a "crawler virus" without knowing it? I'm on a mac and therefor a bit dumb in windoze things.

[fiestagirl]

I know that if someone is using IE and bookmarks your page, there is a box that says "make available offline". This makes the browser visit all of the external links on the page. The UA usually includes "MSIECrawler" when someone does that though.
Sounds like a homemade bot to me.

[Yidaki]

>MSIECrawler

Yeah, due to it's crawling capabilities, i blocked all access to MSIECrawler's.

>Sounds like a homemade bot to me.

That's what i'm pretty sure about too - well, 99.9%! However, the guy is denying it and i thought better asking here, before making him small. ;) You never heard of such a crawler virus? Should i be 100% convinced, that he's lying?

[mincklerstraat]

Sorry for the late reply, just saw this.

I think there's a good chance the guy's innocent, but that he has some weird flukey software installed. I had a similar problem myself - logs showed a whole lot of rogue hits, but not typical of spider behavior - same page being requested zillions of times consecutively. Tried to figure out what the rogue agent was, and in the end I'm pretty sure it was me. I had something installed that, without my knowing it, was making all these zillions of requests. Removed a lot of stuff that had any connection with the browser, and the "rogue agent" disappeared. Agent name was similar to the one you mentioned.