Forum Moderators: open
157.82.157.95 - - [04/Jul/2002:15:51:28 -0400] "GET / HTTP/1.1" 200 3714 "http://m14m.net/hotlist.html" "Steeler/1.1 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)"
That's actually the second hit from them (the first was a request for robots.txt), but this one is more interesting. Unlike most bots, Steeler includes an almost-valid referrer.
I say "almost valid" because my domain's root page isn't linked from that referrer, one of it's subdirectories is. Steeler must be automatically checking the top of any domain it encounters, whether directly linked to or not.
18 requests over the space of 30 hours. Requested everything linked from the root page, including CSS and XML files. Requested the domain root twice with two different referrers, so we know it's not so good at remembering where it's been. That could be problem for some webmasters.
Other than that, it looks like just another research bot, which means somebody here is going to complain about bandwidth threat. Whatever. Their information page [tkl.iis.u-tokyo.ac.jp] says they respect robots.txt and the robots META.
Any thoughts?
If you have a market for far east traffic
I dunno - anything with the name "steeler" (no matter the spelling), "reaper", "harvester", etc. makes me uncomfortable. And Steeler was getting to be bothersome, so off it goes.
