Y!j-bro

is this legit?

[GaryK]

Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; [help.yahoo.co.jp...] YahooFeedSeekerJp/2.0; users 0; views 28)
-----
211.14.9.*
-----
inetnum: 211.8.0.0 - 211.19.255.255
netname: JPNIC-NET-JP
-----
ROBOTS.TXT? Yes
rDNS? No

Quite well behaved. Read and obeyed robots.txt each time over the course of five consecutive days. Only took xml files I use to push out notifications of an update.

The lack of any info to trace this back to Y! makes me suspicious it's a spoof because in the past this crawler has always had a Yahoo-based rDNS.

[GaryK]

Not sure if this is new to you all or not, but the link above leads to a page that seems to identify the user agent prefixes for all of Yahoo's FeedSeekers bots. A friend of mine who is fluent in Japanese confirmed this for me.

Y!J-DSC
Y!J-VSC/ViSe
Y!J-PSC
Y!J-NSC/1.0
Y!J-SRD/1.0
Y!J-BRG/GSC
Y!J-BRI
Y!J-BRE
Y!J SearchMonkey/1.0
Y!J-BRO/YFSJ
Y!J-BRP/YFSBJ
Y!J-BRQ/DLCK

[Pfui]

Thanks and -- good grief. I hope all those respect this:

User-agent: *
Disallow: /

[dstiles]

According to Robtex the sub-range 211.14.0.0 - 211.14.31.255 looks like a server farm. Why would yahoo crawl from that?

In fact there are some yahoo IPs within that range at 211.9.20.0 - 211.9.24.51 - previous and thereafter looks like miscellaneous domains. I didn't check specifically for crawlers. From jpnic it looks as if yahoo "own" that whole block, though it's an odd point to break off...

211.14.20.0/22
211.14.24.0/26
YAHOO-NET
Yahoo Japan Corp.

I've just blocked the range 211.9.0.0 - 211.9.31.255. In fact, I have enough logged baddies from the whole 211.nnn.nnn.nnn range to can the lot but I suspect my customers with Japanese customers would complain. :(