Pfui - 1:27 am on Jun 9, 2006 (gmt 0)

Okay, GaryK. You asked for it:)

1.) Here's a TON of Yahoo-related User-agents (plus fellow travelers) ignoring robots.txt over the past 6-8 months, pretty much excluding most (I hope) of the Yahoo UAs I've already posted about on WW. From umpteen Host names and IPs and countries all pointed toward a single site.

2.) I prefer Apache to show Host names so where you see IPs is where there was no Host used.

3.) Some of these are MyWeb2 (cacheing bookmarker) and phone-related but Yahoo still didn't ask for robots.txt before using a UA like libwww-perl.

4.) The sort is more by Host/IP than not but isn't alpha, sorry, because neither IPs, Hosts nor UAs lent themselves to a sensible listing -- plus there were too danged many of each!

5.) Keeping up with of all of these and still more Yahoo UAs, robots.txt-compliant or not, is really too much work for the traffic Yahoo sends. Way too much.

-----
-----
lj9054.inktomisearch.com
(multi)
Mozilla/5.0 (compatible; Yahoo! Slurp China; [misc.yahoo.com.cn...]

-----
dp131.data.yahoo.com
Mozilla/4.0

rlx-1-2-1.labs.corp.yahoo.com
Mozilla/4.0

-----
r17.mk.cnb.yahoo.com
m23.mk.cnb.yahoo.com
(multi)
Gaisbot/3.0+(robot05[@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)

-----
urlc1.mail.mud.yahoo.com
urlc2.mail.mud.yahoo.com
urlc3.mail.mud.yahoo.com
urlc4.mail.mud.yahoo.com
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

-----
rlx-2-2-10.labs.corp.yahoo.com
Yahoo! Mindset

q02.yrl.dcn.yahoo.com
Yahoo! Mindset

-----
ts2.test.mail.mud.yahoo.com
(68.142.203.133)
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

-----
203.141.52.37
203.141.52.39
203.141.52.44
(multi)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)

203.141.52.47
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

ont211014008240.yahoo.co.jp
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

-----
mmcrm4070.search.mud.yahoo.com
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

-----
proxy1.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1)
proxy1.search.dcn.yahoo.net
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1

proxy2.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Alexa Toolbar; mxie)
proxy2.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

proxy3.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Alexa Toolbar; mxie)
proxy3.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

-----
proxy1.search.dcn.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), libwww-perl/5.69
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

proxy2.search.dcn.yahoo.net
PostFavorites
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90)

proxy3.search.dcn.yahoo.net
PostFavorites
Mozilla/5.0 (Windows; U; Win 9x 4.90; es-AR; rv:1.7.12) Gecko/20050919 Firefox/1.0.7
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051111 Firefox/1.5

-----
msfp01.search.mud.yahoo.com
(side-scroll edited)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0
(compatible; Windows CE; Blazer/4.0; PalmSource; MOT-V300; SEC-SGHE315;
YahooSeeker/MA-R2D2;mobile-search-customer-care AT yahoo-inc dot com)

mmcrm4070.search.mud.yahoo.com
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

opnprc1.search.mud.yahoo.com
Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; [help.yahoo.com...] )

-----
oc4.my.dcn.yahoo.com
YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; [publisher.yahoo.com...]

-----
-----
TROUBLE:

All referers beginning: "http://rds.yahoo.com/"

-----
-----
SPOOFS?

207-234-129-8.ptr.primarydns.com
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

202.139.36.72.reverse.layeredtech.com
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

(Aside: .layeredtech.com (now 403'd) routinely sends forth plagues of bots. Six different kinds just last month alone.)

###