GaryK - 9:33 pm on Jun 8, 2006 (gmt 0)

I will ask them Bill. But what they really want is a list of user agents that we know have disrespected robots.txt.

This is what I've managed to find on my own and the threads where I found them:

[webmasterworld.com...]
Yahoo! Mindset
66.228.182.177
66.228.182.183
66.228.182.185
66.228.182.187
66.228.182.188
66.228.182.190

[webmasterworld.com...]
Mozilla/5.0 (compatible; Yahoo! DE Slurp; [help.yahoo.com...]
72.30.142.24

[webmasterworld.com...]
Mozilla/4.0
66.228.173.150

[webmasterworld.com...]
YRL_ODP_CRAWLER
Sorry no IP Address but rDNS: rlx-1-1-1.labs.corp.yahoo.com

[webmasterworld.com...]
mp3Spider cn-search-devel at yahoo-inc dot com
202.165.102.179