Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: martinibuster
Yahoo! will be modifying its web-indexing robot (crawler) user-agent to reflect our company name. This is the crawler we use to build a searchable index for the Yahoo! search services. To ensure consistency and minimal disruption, we will continue to maintain the "Slurp" name within our web crawler user agent and continue to support "Slurp" as part of any robots.txt files that references this.
Yahoo! Slurp will continue to obey the Robot Exclusion Standard (http://www.robotstxt.org/wc/exclusion.html).
The complete User-Agent that will be logged as part of this transition will be shown in your logs (if user-agent logging is enabled) as follows:
Mozilla/5.0 (compatible; Yahoo! Slurp; [help.yahoo.com...] )
For more information on the Yahoo! Slurp user-agent, robots.txt usage or accesses please refer to the slurp information page now located at:
This transition will take affect starting Monday Feb 16th, 2004.
I agree with their reason for keeping the Slurp name. Backwards compatibility, it's everyone's friend.
Why's it brave? Slurp was hardly a prestigious spider. I think it was pretty dumb and fairly cowardly; especially for query strings. I'm shooting from the hip but I suspect the most intelligent spider Yahoo! bought on their search-engine-fest was FAST. I'd see Fast Yahoo! as a more serious threat to Googlebot, but although I shouldn't judge the new spider by its name, I still snigger when I see Slurp's user agent.
Yahoo Shopper, of course, has been bimbling around robotland for a while now.
(Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)
that resolves to mmscrm03-1.sac.overture.com and the whois says overture.com;
but I also get -
Network id#: 1
Qwest Cybercenters QWEST-CYBERCENTER-2 (NET-66-77-0-0-1)
22.214.171.124 - 126.96.36.199
Fast Search, Inc. QWEST-MCC-FASTSRCH3 (NET-66-77-73-0-1)
188.8.131.52 - 184.108.40.206
ARIN WHOIS database, last updated 2004-02-19 19:15
is this a new crawler that's using a FAST/Overture combo of some sort..?
On the other hand, the Yahoo Seeker spider is spidering deeply.
Multimedia crawler for Alltheweb, AV and other Overture partners Index.
Hi Tim, it's great to have you on the forums.
Is this a fair assessment?
+ Scooter still finds results for AV and AV is still a stand alone search engine in its own right.
+ FAST still finds results for AtW and AtW is a still a stand alone search engine in its own right.
However, new technologies developed by Yahoo! are likely to be single items shared by all the search engines owned by the company? Yahoo-MMCrawler is an example of this?
I think the new Yahoo results look really good. The web page summaries are especially fair.