Pfui - 7:03 pm on Sep 11, 2011 (gmt 0)
1.) Mokita: Good thought about checking robots.txt. Mine's checks out A-OK. It's also CGI-generated so I know exactly what rules Yahoo/Slurp gets.
2.) Ironically, even "Yahoo! Slurp China" --
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
-- requests robots.txt, although it always ignores it. (So it gets 403'd for every file other than robots.txt.)
And then there's the Yahoo UA that never requests robots.txt, only favicon.co --
I get maybe 10 Yahoo-referred hits a month, and most are to two 'answers.yahoo.com' replies with links, not SERPs per se. Thus there's precious little benefit I can see in allowing anything other 'plain' "Yahoo! Slurp" from ".crawl.yahoo.com" access to anything other than .html files. YMMV