Pfui - 10:15 pm on Sep 28, 2011 (gmt 0)

- Did it read, and heed, robots.txt?

- Any rDNS data for any single IP?

- I don't know if that UA is new, or old, or a new name for a new, or old, hybrid, or what. Just that its name recalls a number of UAs Yahoo spawned in recent years, dating back to when Y was very, very picky about UAs being specifically named in robots.txt so we included them all...

User-agent: YahooMobile
User-agent: YahooCacheSystem
User-agent: Yahoo! Slurp/Site Explorer
User-agent: Mozilla/4.05 [en]
User-agent: LTI/LemurProject
User-agent: Yahoo-Blogs
User-agent: Yahoo-Blogs/v3.9
User-agent: Yahoo-MMCrawler
User-agent: Yahoo-MMCrawler/3.x
User-agent: YahooYSMcm
User-agent: YahooYSMcm/2.0.0
User-agent: Yahoo-Test
User-agent: Yahoo! Mindset
User-agent: Y!J-BSC
User-agent: Y!J-BSC/1.0
User-agent: Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: y!j-bsc
User-agent: y!j-bsc/1.0
User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: Y!J
User-agent: Y!J/1.0
User-agent: Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: y!j
User-agent: y!j/1.0
User-agent: y!j/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
User-agent: Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)
User-agent: Mozilla/4.0 (compatible; y!j; for robot study; keyoshid)
User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
User-agent: Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
User-agent: Mozilla/5.0 (Yahoo-Test/4.0 mailto:vertical-crawl-support@yahoo-inc.com)
Disallow: /

(Hmm. I could probably just cut all those lines because now I only allow User-agent: Slurp (w/ specific rules) and mod_rewrite/whitelist (or blacklist) everything else from Y. It's --

User-agent: *
Disallow: /

-- or bust:)