Y!j-bsc/1.0

New Yahoo spider?

[bobothecat]

203.141.52.37 - - [28/Feb/2006:00:28:01 -0700] "GET /robots.txt HTTP/1.1" 206 4695 "-" "Y!J-BSC/1.0 [help.yahoo.co.jp...]

The IP does resolve to Yahoo Japan

[GaryK]

[webmasterworld.com...]

[Pfui]

If you have more time/interest in Things Yahoo, check out the Yahoo Search [webmasterworld.com] forum here, too. I steer clear because I'm weary of tweaking robots.txt and .htaccess for every new Yahoo crawler sighting:)

Slight Tangent Alert --

FWIW, I'm not wild about any of Yahoo's Asian crawlers (Y!J and Slurp China) because I already have firewall-worthy boatloads of trouble coming in from those parts of the world.

So as an example of 'If it crawls, stomp on it' robots.txt over-writing, here are the almost silly permutations I'm Disallowing in addition to a blanket...

User-agent: *
Disallow: /

...because of the uppercase and lowercase differences showing up or being reported. (If I could only read the non-English reference URLS, or if Yahoo put the non-U.S. crawler info on yahoo.com, I could cull this crazy list:)

## YAHOO CRAWLERS/URLS
## Slurp: [help.yahoo.com...]
## HOST: .inktomisearch.com; .mail.mud.yahoo.com
## Slurp CHINA: [misc.yahoo.com.cn...]
## Blogs: [help.yahoo.com...]
## MM: mms dash mmcrawler dash support at yahoo dash inc dot com
## Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

User-agent: Yahoo-Blogs
User-agent: Yahoo-Blogs/v3.9
User-agent: Yahoo-MMCrawler
User-agent: Yahoo-MMCrawler/3.x
User-agent: YahooYSMcm
User-agent: YahooYSMcm/2.0.0
User-agent: Y!J-BSC
User-agent: Y!J-BSC/1.0
User-agent: Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: y!j-bsc
User-agent: y!j-bsc/1.0
User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: Y!J
User-agent: Y!J/1.0
User-agent: Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: y!j
User-agent: y!j/1.0
User-agent: y!j/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)
User-agent: Mozilla/4.0 (compatible; y!j; for robot study; keyoshid)
Disallow: /

##

FWIW redux --

I recently relaxed the reins on Yahoo, and Slurp, to see if I'd gain more traffic. Unfortunately, all I've gained is more work figuring out why in the heck Yahoo uses so many different crawlers with so many different names and so may different behaviors. Amazingly, the above are just the ones I think/hope will read robots.txt. I control still more using mod_rewrite. Enough already!