Search Engine Spider and User Agent Identification Forum

Thanks for the feedback, and for the actions your teams have taken.

Visits from clients at proxyn.search.dcn.yahoo.net or proxyn.search.acd.yahoo.net are not crawler activity but proxies for browser page views using a 'translate this page' link from Yahoo! search results. Babelfish provides the automated translation. The Babelfish team will be modifying their proxy headers to more accurately reflect the page access.

This kind of agent (language and markup tranlator) is the most problematic. Because most webmasters only have access to the HTTP headers available in NCSA extended/combined log format (Referer and User-agent), I'd argue in favor of 'injecting' an additional parameter into the translation proxy user's User-agent string, or modifying the Referer header. This because most webmasters won't even be aware of the proxy-related headers like "HTTP_VIA", "HTTP_X_FORWARDED_FOR", etc. -- or have the knowledge or capability to test them.

We had a discussion here years ago with a spider author who was upset about being massively banned, because he thought he'd done his due-diligence by providing identifying and contact info in the "HTTP_FROM" header. But the fact is that almost no Webmasters can see that header in their logs; It requires a custom log format, and that level of configuration is simply not available on common name-based virtual hosting accounts.

So anyway, I'll toss that UA-string-injection idea out for your team to consider: Find the last substring in the user's UA string bounded by ";" and ")", insert "; Yahoo! Babelfish/1.0" ahead of the ")" and you're good to go. (Of course, that assumes that the user's UA string is valid to begin with.) I'm suggesting this only if it doesn't make more sense to modify the referer, and only for 'translatable objects', not for images, stylesheets, and client-side scripts, which I presume just pass through the translation proxy unmodified.

The 'Mozilla/4.0' UA from Overture Services (like 66.228.173.150 as described in [webmasterworld.com...] ) is an agent for editorial checking from Yahoo! Search Marketing (Overture). This editorial agent only reads URLs submitted by advertisers for sponsored search listing by Yahoo! Search Marketing (YSM); it does no extraction and is not a public content crawler. Information about the editorial agent is in the terms of service documents for advertisers listing with Yahoo! Search Marketing. [...] This YSM agent was updated the week of June 5 to rate limit its activity better after an advertiser does a batch submit.

I'm not familiar with the 'environment' in which this Overture User-agent operates -- To be blunt -- whether you can 'trust' the target sites. But for the majority of activity, I really come down on the side of those who say that all automated agents should identify themselves, even if just with a "(compatible; Yahoo! Sponsored Search)" or "(compatible; Overture)" in there. The plain "Mozilla/4.0" User-agent is commonly used for site scraping and other exploits, and is persona non-grata on many sites. Yahoo can use it on my sites, but only because the blocking rule contains IP and remote host exclusions to let Yahoo! in. So, I'd say identify yourself, spot-check for underhanded UA-based cloaking (if necessary) using a valid browser UA, and drop any subscriber that breaks your TOS. Rule of law and all that...

I hope these comments are useful. Thanks again to you and your teams for listening to feedback and taking action on our concerns.

Jim

I'm 100% behind this sentiment as there are ZERO reasons that we have to subject ourselves to more than one Yahoo! crawler. They should all share a common data set internally and not continue to burn up our bandwidth over and over and over.

If that common data is too old for the task that wants it, fine, queue it up for a refresh crawl, but only ONE refresh crawl, not FIVE.

BTW, you didn't explain this:

Thank you! That would be a wonderful relief. One of our sites has 110 pages available for indexing, but only 18 pages are likely to change - the rest are static information pages or product pages which don't normally change once they've been posted. Crawling those once per month would be quite sufficient for freshness.

In the first 14 days of this month the Yahoo! Slurp bot has requested 737 pages and robots.txt 449 times - roughly 32 times per day. This is really taking freshness to the extreme! Compare Slurp figures with Googlebot, which in the same period has requested 172 pages and robots.txt 16 times - slightly more than once per day. I am perfectly happy with the freshness of our results in Google.

Does Slurp take any notice of the "revisit-after" metatag?
<meta name="revisit-after" content="30 days">

66.228.165.143 - - [15/Jul/2006:15:54:12 -0600] "GET / HTTP/1.1" 301 344 "-" "Mozilla/4.5 [en] (Win98; I)"

Naughty Yahoo User Agents
[webmasterworld.com...]

Addressed in his response are:

Yahoo! China
Yahoo! Mindset
proxyn.search.dcn.yahoo.net
proxyn.search.acd.yahoo.net
Mozilla/4.0 (Overture).

.
2.) Alas, still unaddressed are the majority of entries I listed in what is now message "#:400187" including the following (with still more new ones, below; plus cut-to-the-chase stats in #6):

>>
-----
dp131.data.yahoo.com
Mozilla/4.0

rlx-1-2-1.labs.corp.yahoo.com
Mozilla/4.0

-----
r17.mk.cnb.yahoo.com
m23.mk.cnb.yahoo.com
(multi)
Gaisbot/3.0+(robot05[@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)

-----
urlc1.mail.mud.yahoo.com
urlc2.mail.mud.yahoo.com
urlc3.mail.mud.yahoo.com
urlc4.mail.mud.yahoo.com
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

-----
ts2.test.mail.mud.yahoo.com
(68.142.203.133)
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]

-----
203.141.52.37
203.141.52.39
203.141.52.44
(multi)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)

203.141.52.47
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

ont211014008240.yahoo.co.jp
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)

-----
mmcrm4070.search.mud.yahoo.com
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

-----
proxy1.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1)
proxy1.search.dcn.yahoo.net
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1

proxy2.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Alexa Toolbar; mxie)
proxy2.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

proxy3.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Alexa Toolbar; mxie)
proxy3.search.scd.yahoo.net
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 4.0), Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)

-----
msfp01.search.mud.yahoo.com
(side-scroll edited)
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0
(compatible; Windows CE; Blazer/4.0; PalmSource; MOT-V300; SEC-SGHE315;
YahooSeeker/MA-R2D2;mobile-search-customer-care AT yahoo-inc dot com)

mmcrm4070.search.mud.yahoo.com
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)

opnprc1.search.mud.yahoo.com
Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; [help.yahoo.com...] )

-----
oc4.my.dcn.yahoo.com
YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; [publisher.yahoo.com...]

-----
All referers beginning: "http://rds.yahoo.com/"
<<

.
3.) And here's another Yahoo Host/UA that never asks for robots.txt during its weekly visit:

morgue2.corp.yahoo.com
Mozilla/4.05 [en]

And what's this new referer? "http://yq.search.yahoo.com/"

.
4.) And then there are these new -- licensees? Fakes? And again, no robots.txt by either:

www.io.com
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]
07/06 09:36:11 /

phad.cc.umanitoba.ca
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...]
07/15 12:19:20 /

.
5.) I try to cooperate with Yahoo but I'm repeatedly abused by them, by their scores of obvious and covert bots and UAs and IPs, by their retrieving info and licensing it -- e.g., thumbnails to Viewpoint -- info that they're not supposed to retrieve in the first place.

Where's the Help page with the code to prevent a thumbnail grab? Because apparently this list missed the little sucker:

Yahoo! Slurp China has just violated our robots.txt, which contains both the following entries (to be sure one of them will work!)

User-agent: Yahoo! Slurp China
User-agent: Slurp China
Disallow: /

Logged:

lj910157.inktomisearch.com - - [18/Jul/2006:14:08:13 +1000] "GET /robots.txt HTTP/1.0" 200 1815 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp China; [misc.yahoo.com.cn...]
lj910193.inktomisearch.com - - [18/Jul/2006:14:08:23 +1000] "GET / HTTP/1.0" 403 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp China; [misc.yahoo.com.cn...]

Luckily I still had mod_rewrite blocking it.

Yahoo_Mike: Please would you confirm what is the correct syntax to successfully block Yahoo! Slurp China via robots.txt. Thanks.