Willow Internet Crawler

[Pfui]

This bot's been around a lonnng time, and it's been a lonnng time since I've seen it. Circa 2003 threads [webmasterworld.com...] [webmasterworld.com...] suggested it filtered k-12 content.

But judging from the following instant-triplet visitors from three different states, school-related anything looks iffy to me. I'm including unobfuscated Host/IP details because Project Honey Pot pages show prior hits using --

Willow Internet Crawler by Twotrees V2.1

-- for all of these:

24-119-250-19.cpe.cableone.net [projecthoneypot.org...]
17:28:21 /

67.217.153.187 [projecthoneypot.org...]
17:28:24 /

66.99.209.4 [projecthoneypot.org...]
17:38:39 /

robots.txt? NO

[keyplyr]

From today:

24.117.227.227 Willow Internet Crawler by Twotrees V2.1"

I've always blocked it. See it about 3xs a month.

[Pfui]

More hits last night, from one with Willow bot-running history and one without. And then along comes a third hit, from the mothership:

bell-65-68-14-194.pool.twotrees.net
Willow Internet Crawler by Twotrees V2.1

robots.txt? NO
IP = 65.68.14.194 [no Project Honey Pot hx]
robtex: "twotrees.com has one IP number (68.224.164.10), but the reverse is www.twotrees.net"

twotrees.net promos their reselling hardware and software to schools, plus monitoring network activity. But none of the maybe-monitoring hits I've seen were school-based and most weren't even during (U.S.) school hours/days so I'm not sure what they're really doing or why. In any event, "crawler" suffices.

(Too bad the company isn't up front about its crawler, and that its crawler doesn't respect robots.txt, because the crawler-denied content is educational and suitable for teens.)

[dstiles]

I've seen it once this month (can't say off-hand about previous months) from AT&T. It hit a wholesale groceries site, primarily UK/Euro.

The bot kills itself so no real problem.