Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Inclusion Program

Trying to clear up some misunderstanding in this forum



10:02 pm on Feb 27, 2004 (gmt 0)

10+ Year Member

The primary means of generating our index is via our free crawl, using our new Yahoo! Slurp crawler. Yahoo! Slurp discovers pages by following links on the web. We update our index with a daily crawl to gather newly created and fast-changing URLs, as well as our main crawl which updates our index incrementally twice per week.

We supplement this free crawl with the paid inclusion program, which allows us to add dynamic database content and other content that we could not otherwise discover and crawl. Currently less than 1% of our index is content that is included via our inclusion programs.

Our paid inclusion programs provide a superior way for sites to interact with us in a clearly structured fashion. These sites get the following
1) Control over which URLs they include in the index (subject to content
2) Frequent refresh
3) Clearly defined way to interact with us (for example: clear feed
4) Quality review and consultation (for feed customers)
5) Detailed reporting to track and optimize performance
6) Customer service

Participation in Yahoo!'s paid inclusion program does not guarantee rank in the search results. URLs are ranked in search results in the same way as all other URLs from the free crawl. The key drivers of ranking are keyword-specific relevance and site quality, as assessed by our regular search relevance algorithms with input from the quality review process.

(Note that Inktomi Index Connect clients are receiving Yahoo! Search traffic going forward, and Inktomi Search Submit customers are receiving a free trial of Yahoo! traffic until 4/15/04.)

We will also be launching free URL submission during the week of 3/1. This service allows content providers to suggest that the crawler should visit a site during the main crawl cycle. This service is different than paid inclusion in two ways:

1) There is no guarantee that we will crawl and include the URL in the index.
2) As this site potentially could be added to the main crawl, Yahoo! Slurp will follow links from that URL.

This service will reside on the Yahoo.com site and be accessible via a Yahoo! login. This service has been available for the AltaVista and Alltheweb Indices previously and the tradition will continue with Yahoo! Search.


2:11 am on Mar 3, 2004 (gmt 0)


SiteMatch is an incremental erosion to the free traffic that Yahoo once provided. It is a misguided attempt at increasing their short term revenues at the cost of long term search engine user satisfaction.

Tim or some other members will probably talk about 'free crawling', but if you do a search on certain keywords you can already see that free SERPs have been pushed below the fold.

Do not let anyone kid you - Yahoo is incrementally moving forward in the full monetization of all search engine traffic.


2:12 am on Mar 3, 2004 (gmt 0)

10+ Year Member

i agree with you no carrier but in many business instances a 30˘ click will be much lower than a yahooverture click. Thats about it. If you optimize correctly you might not be in the first 3-5 positions as they are taken up with PPC but you will be on the first page. I am NOT on Yahoovertures side as you can see by what i call them. But time marches on and we have to learn how to sWebmasterWorldith the sharks.


2:53 am on Mar 3, 2004 (gmt 0)

10+ Year Member

Here is a question - maybe Tim can answer.

Assuming you are signed up presently for the Inktomi PPI service which is going away .....

Is there any commitment from Yahoo that *ALL* of these URL's / Listings will be crawled *AT LEAST ONCE* - to be included in the free searches - prior to the 04/15 discontinuance of Inktomi?

If so - this might answer some of the angst you hear expressed here. Since users feel we paid to be in Yahoo when we signed up for Inktomi (I would not have signed up at all, except for Yahoo) - and theoretically there is no preference in Yahoo given by the "Relevancy engine" - as stated by Tim - for the new Site Match program - then theoretically - your present ranking would not change. At least - for what is already in the index - and paid for.


So - Tim - what is the possiblity of Yahoo making this commitment to your somewhat vocal legion of Inktomi PPI subscribers?


2:56 am on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I was doing a test run of Site Match. It felt like a deja vu of Looksmart. The form was even screwing up in the same places as the Looksmart form. It was an omen to me.

Oh, by the way Yahoo some of these PPC engines mention there is a minumum deposit with how much it will cost. They don't wait until you're half-way in the sign-up process to bring it up. Oh, I know I'm picky, picky, picky, but I'm fresh off being hood-winked by Looksmart and Positiontech. They tell me there is a little bit of dishonesty on the Internet.


3:15 am on Mar 3, 2004 (gmt 0)

Marcia you are resorting to vague, general statements which are not even addressing the topic at all.


1. The overture release does not refer at all to free crawling.

2. The erosion of free results has already occured with PPC - try searching on competitive keywords and you will see free results below the fold only.

3. The addition of SiteMatch is a step towards charging for traffic rather than giving it away for free.


Yahoo is incrementally eroding free SERP traffic.

There is nothing kindergarten about this.


3:40 am on Mar 3, 2004 (gmt 0)

10+ Year Member

In the end we will all have to wait and see what Yahoo does. I'm willing to give Yahoo the benefit of the doubt here, until I have reason to do otherwise.

I would also point out that Tim himself - he in no wise is obligated to be on this forum. Or answer any of our questions. Or to withstand any of our abuse or frustration. He came on this forum of his own free will, to help us all - and provide us with early and valid information. For this we owe *him* personally our thanks - even if we become frustrated with Yahoo. Don't kill the messenger - it serves no purpose. I would rather have a valid and reliable source of information - even if I don't like the message. I think we will all agree on this point....

If Yahoo does

1) Launch a free submission (assuming updates of a sane period of time - say - monthly).

2) Do not give a relevancy edge to the PPI folks, as promised already by Tim ....

3) Execute an Inktomi crawl, and include those results prior to the disappearance of Inktomi, such that the Inktomi PPI folks don't feel cheated, and their existing results remain more or less intact.

4) Do not implement any sort of penalty in the free searches - assuming you sign up for a search term / URL in the new program - and then discontinue the pay-per-click once the free result comes online.....

Then this new update could be a boon for folks paying $1.00 per click on some of the more competitive Overture keywords. Allowing them to be on many more search phrases, at the same cost, and increase their sales. Dropping their advertising cost per sale.

They could submit a URL on this new program, be operational in 48 hours - and when their free search kicks in - back off the pay-per-click. I would pay $49 a month and .15 cents a click - for one month - while the index cycles in many cases. Just so long as the free searches rank equally high, and I can drop the paid search without penalty.

I reiterate my question to Tim - will Inktomi be crawled by the free spider - and results merged - prior to the disappearance of Inktomi?

If so - and the above is true - there may not be such a hot topic.

And Tim - if you would be so kind - please comment on all of the items 1 through 4 above.

I will patiently await Tims reply.


3:44 am on Mar 3, 2004 (gmt 0)

WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Yahoo is incrementally eroding free SERP traffic

Of course... and the point is...? Logiclamp, this seems to be a rather circular discussion. Yes, Yahoo has heavily monetized results. They have been that way for a while - a couple of years ago I taught an e-commerce class and used a slide showing Yahoo's entire first screen being occupied by various kinds of paid results. The new program may further increase the paid real estate.

I would expect Yahoo to maximize its search revenue, keeping in mind that if the results are worse than Google's free results Yahoo will lose market share over time. That's why they switched to Google in the first place - they were losing share because Google results were better.

Unfortunately, there's no Bill of Rights guaranteeing free search results. It's a user choice - if they get better results from search engines with fewer paid results, they'll gravitate to those engines.


3:46 am on Mar 3, 2004 (gmt 0)

rogerd - can I quote you? There are some moderators and members around here who think I am crazy for saying that.

yes it is obvioius, that is why I shake my head in wonder everytime someone disagrees with what you are saying.


4:11 am on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I think everything being posted in these Yahoo forums is positive. Its only natural for people to test the staying powers of a new program and the people behind it. People don't want to think they're being led blindly down some PR path. In the past year a lot of people have lost money with Google and others. They're demanding more honesty of themselves and companies on the Internet. I like what I'm seeing and the questions being posed.


5:22 am on Mar 3, 2004 (gmt 0)

10+ Year Member

Have the FTC implications of this been discussed?

This might barely meet the request but may not:

SPONSOR RESULTS () (Become a Sponsor)

And this certainly looks deceptive under the guidelines:

TOP 20 WEB RESULTS out of about 7,150,000 (What's this?)

[edited by: quotations at 6:52 am (utc) on Mar. 3, 2004]


6:32 am on Mar 3, 2004 (gmt 0)

10+ Year Member

I think the program reflects all the internal rangling that must go on at Yahoo search. It isnt too easy to integrate cultures of different acquired and merged companies.

The result here is that Yahoo has designed a camel - a horse designed by a committee.

Appears obvious that too many people at Yahoo were involved in coming up with this program.

It may be great for the SEO industry, as its another product to sell but there are way too many serious issues with it.

Yeah.. I know people will say its too soon to see but good marketing requires good up front communication. Yahoo's launch of this program to the SE community appeared to be just as disjointed as the program itself.

What has Yahoo accomplished?

The waters are now muddied at Overture. Used to be easy to understand what Overture did.

LOL why didnt Yahoo just create or buy another company to do run this program .... theyre like a kid collecting baseball cards til they dont know what they have in their collection.


7:02 am on Mar 3, 2004 (gmt 0)

To everyone who does not believe Yahoo will be providing free submissions, and free listings, stop and give them the benefit of the doubt. Yahoo, through Tim and whoever else, is closely monitoring everything on this board, and other ones Im sure. They know the hell webmasters will raise if the free listings do not continue. I believe every word of what Tim is saying. I find it illogical for them not to offer free listings, when millions of quality websites have no ecommerce intention, and are simply there providing quality information.


7:32 am on Mar 3, 2004 (gmt 0)

10+ Year Member

Site Match seems simple to us…we now pay for listing we were getting for free thus increasing our costs yet increasing Yahoo’s short term profits. I wonder how many lost hours went into the brainstorming of this program. And I bet that within six months to increase revenue, they’ll increase the per click fee and/or the listing fees.

If Yahoo really wants to impress me, how about a program where I’m only charged for a click that results in an actual sale? I’ll even pay a commission fee per click for a program like this. Site Match is basically a low-profile old style banner program…high costs yielding very little return. At least for us, a high majority of people Yahoo drives to our site never results in a sale.

The point is that if I have to pay Yahoo (and other similar companies) so much money to drive people to my site, I now want quality leads and guarantees that a certain percent of clicks will result in sales. I know this is asking a lot and would be a logistical nightmare for Yahoo but it’s long term thinking, something that they are not prone to do.

With all these changes going on the search engine market is very ripe for something new and innovative. A new search engine that’s doing unique things could easily capture a lot of market share from Yahoo and Google. It will be interesting to see if anyone makes a move by the end of the year.


7:56 am on Mar 3, 2004 (gmt 0)

10+ Year Member

we now pay for listing we were getting for free

No you are not. Have you read what Tim has been writing in this thread?


8:01 am on Mar 3, 2004 (gmt 0)

10+ Year Member

Why has Yahoo left out so many top quality sites?

They keep saying, we want to spider as much content as possible yet there is a large group of quality sites that do not show in their results.

It isnt that Yahoo's spider hasnt found these sites..as they are indexed but just dont show up in search results unless the specific domain is searched on.

Yahoo's constant repetition that they are getting as many sites listed as possible is bs.


8:02 am on Mar 3, 2004 (gmt 0)

10+ Year Member

Why has Yahoo left out so many top quality sites?

And Google and MSN and Ink and .... don't do that?


8:49 am on Mar 3, 2004 (gmt 0)

10+ Year Member

There's a touching degree of naivety here from a few people: a belief that Yahoo! will listen to webmasters - and a belief that Yahoo! will continue to provide meaningful free traffic.

Why? Yahoo! has never listened to webmasters before. Why should it start now? And you can see that the "free" traffic is going to be pushed so far down the SERPs that it won't matter if you're listed or not.

I'm still laughing at this one though. The deal is that I have to spend several thousand dollars just to get my site listed in Sitematch and then I have to pay for every click whether or not that click is actually relevant to my site. There is no way to control costs on this at all.. at least with Google I know that they've typed in a relevant search phrase. At least with Inktomi I know that my cost exposure is limited. So why should I put money into this when I can get more relevant results from Google, cheaper? Has Yahoo! gone nuts?


9:16 am on Mar 3, 2004 (gmt 0)

10+ Year Member

After reading todays newsletter about Overture's Site Match & Site Xchange program.
I have a few questions in mind that is inline with this subject.

1. How long does the review take of a particular site for the Xchange program.

2. Those who have paid for Ink inclusion a month or few months back will be deactivated permanently in the Yahoo index once the program is in place. But will it still work in MSN?

3. So can Tim confirm as he mentioned earlier that the program will start on April 15th?


Would also like to know when yahoo updates their index, I've been spidered for the past month & none of my sites appear yet, oops yahoo! slurp just started on the 18th - so when do you guys update?

The serps does looks quite bad over here for certain search phrase... really out of joint - no quality sites


9:56 am on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member

I have heard that those URLs reviwed via Site Match will be given a little cross in a box somewhere. The only other sites to be given the same treatment will be those chosen by Yahoo! employees.

I wonder what effect this will have on Site Match URLs. Will they rise to the top of the Yahoo! SERPs as they are deemed more relevant?


10:00 am on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

I don't think you want it explained to you logiclamp, but the program isn't for you. It's for Amazon to get thousands of very deep product pages into the database in a timely fashion.

Most sites will be crawled for free and will have no reason to join the program.

What's wrong with that?


12:48 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

I talked with Jeff Weiner the senior VP of search at yahoo last night. He was adament that they are going to crawl the entire web for free. Their intention is to find every public page available and put it in the index.


1:00 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

That's encouraging news Brett.

AzCowboy - you talk about Inktomi finishing April 15. My understanding from Tim's comments is that isn't the case. Inktomi will still be operating See message 59

2. Customers who have already paid for the current Inktomi program will continue to get all the benefits they have paid for - distribution across the pre-Yahoo! Inktomi network. As an additional bonus, they will get 60 days of free traffic on the new Yahoo! distribution.

Of course the only real value in the "pre-Yahoo! Inktomi network" is MSN & how long they'll stick with Ink is the $49 question...


2:54 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member caveman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

I talked with Jeff Weiner the senior VP of search at yahoo last night. He was adament that they are going to crawl the entire web for free. Their intention is to find every public page available and put it in the index.

Very reassuring, thank you!

Since we're not there yet, however (there are still many large gaps in the SERP's apparently related to INK's prior spotty approach), any sense of when we might see more evidence of this?

I have a feeling it's the lack of evidence so far that is still causing some concern - e.g., we have several sites where only the paid pages show well, while the unpaid are still often not showing at all, even though the lost pages are parallel in structure, importance, etc. to the paid/showing pages...


2:55 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Pay inclusion especially at the prices mentioned on Overture just doesn't make sense. It virtually would elimate all non-commercial sites and only allow for very large commercial sites. I couldn't afford 30 cents a click and/or the inclusion costs. I hope Brett you're right. I worry that these companies forget that the majority of us don't have multi-million dollar ad campaigns.


3:02 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

PLEASE somebody tell me how to stop the emails I asked for on this thread?


3:16 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

I think that some of us are forgetting that for Yahoo to be successful and competitive in the search world they will need to crawl and index ALL web pages and rank them according to search relevancy and page/site quality, otherwise the searchers will go to search engines that do. In order to provide this kind of service Yahoo will have to provide free crawl and indexing services. Having said that, Yahoo is a business and has to be profitable to stay in business. To do this they have no choice but to sell ads or charge for special services such as frequent crawling and indexing.


3:28 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I don't understand why this is so important to everyone.

If this were a few years ago before all of the overture listings at the top of the page, I'd see the point. Yahoo regular SERPs don't deliver the amount of traffic they used to.

There are a few different scenarios:
1) keywords not competitve. No (or little) Overture, no need to pay PFI, simple optimization should bring you to the top.

2) keywords moderately competitve. Overture listings will take almost all of the searches. Few people will scroll down to regular SERPs. No need to pay PFI.

3) keywords highly competitve. Overture listings take most of the searches, but still in absolute numbers a lot of people venture to the regular SERPs. If the words are so competitive, the margins should be high and PFI should be a bargain.


5:40 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Brett - Thanks for sharing that nugget. It's a pretty big nugget.


6:26 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Their intention is to find every public page available and put it in the index.

Yes but where will a free crawled listing end up in the index?


6:39 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

they are going to crawl the entire web for free. Their intention is to find every public page available and put it in the index.

I think people are forgetting this claim...Remember you are paying Yahoo the setup fee and CPC to guarantee a frequent crawl. If you don't want to pay for site match you'll be indexed too, for free. You just won't get as many chances to optimize your pages as a person would using Yahoo's PFI system.

Do people already forget the stress that we all associated with the montly google dance? Imagine if yahoo went into a monthly update like this? We'd all lose even more sleep. This PFI system allows people to bypass the stress associated with infrequent updates/crawls.

Yahoo is providing an added service to webmasters that they are free to use or not use (I don't think webmasters should feel obligated in any way to use this program since they won't lose out on much if they don't)

Anyways, that's just my 2 cents and I am happy yahoo is providing this service.

This 423 message thread spans 15 pages: 423

Featured Threads

Hot Threads This Week

Hot Threads This Month