Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Inclusion Program

Trying to clear up some misunderstanding in this forum



10:02 pm on Feb 27, 2004 (gmt 0)

10+ Year Member

The primary means of generating our index is via our free crawl, using our new Yahoo! Slurp crawler. Yahoo! Slurp discovers pages by following links on the web. We update our index with a daily crawl to gather newly created and fast-changing URLs, as well as our main crawl which updates our index incrementally twice per week.

We supplement this free crawl with the paid inclusion program, which allows us to add dynamic database content and other content that we could not otherwise discover and crawl. Currently less than 1% of our index is content that is included via our inclusion programs.

Our paid inclusion programs provide a superior way for sites to interact with us in a clearly structured fashion. These sites get the following
1) Control over which URLs they include in the index (subject to content
2) Frequent refresh
3) Clearly defined way to interact with us (for example: clear feed
4) Quality review and consultation (for feed customers)
5) Detailed reporting to track and optimize performance
6) Customer service

Participation in Yahoo!'s paid inclusion program does not guarantee rank in the search results. URLs are ranked in search results in the same way as all other URLs from the free crawl. The key drivers of ranking are keyword-specific relevance and site quality, as assessed by our regular search relevance algorithms with input from the quality review process.

(Note that Inktomi Index Connect clients are receiving Yahoo! Search traffic going forward, and Inktomi Search Submit customers are receiving a free trial of Yahoo! traffic until 4/15/04.)

We will also be launching free URL submission during the week of 3/1. This service allows content providers to suggest that the crawler should visit a site during the main crawl cycle. This service is different than paid inclusion in two ways:

1) There is no guarantee that we will crawl and include the URL in the index.
2) As this site potentially could be added to the main crawl, Yahoo! Slurp will follow links from that URL.

This service will reside on the Yahoo.com site and be accessible via a Yahoo! login. This service has been available for the AltaVista and Alltheweb Indices previously and the tradition will continue with Yahoo! Search.


7:48 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Paid Inclusion is dead, now it's Paid Inclusion CPC. For those that it makes economic sense, do it, but for those of us who really just have a few pages they liked to keep on top of with the ability to refresh every 48 hours, can a separate, true Paid Inclusion program be created, with a flat annual fee? Maybe the customer support part can be scaled back? Maybe the inclusion fee structure can be changed and capped, but the CPC part rules me out, in effect transitioning me from a paying customer to a free content provider.


8:09 pm on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

>>>crawl the entire web for free.<<<

All search engines crawl the web for free. Crawling and indexing are two completely different things.

>>>Their intention is to find every public page available and put it in the index.<<<

Does public page equal .edu and .gov or does that mean that will include amazon.com or edmunds.com in their free index? Could someone (from Yahoo)define "public page"?

Sounds like if you want to buy an apple pie, Google is the way to go - but if you want the safety rating on a child seat --- well I'd still use google because I like being the one to decide what's valuable to me and what's not.



8:33 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

I have a travel destination guide Web site that provides information to those planning to visit my area. I do not charge travel related businesses for providing information on their business nor do I run paid ads. The total income from this site comes from Google Adsense ads. If I were to use CPC advertising, the cost would be more than what I make from the Adsense ads. My only options are free or PFI listings. I would welcome the opportunity to publish Yahoo "Adsense" type ads if the didn't conflict with Google ads or if they were more profitable.


8:49 pm on Mar 3, 2004 (gmt 0)

There's no question about it...there will be free crawl and inclusion.

The big question is...

Will the free inluded URL be treated the same as the paying URL?

If not then your free URL inclusion is just actually helping Yahoo to make money out of it in exchange for mediocre traffic coming from them.

Okay, let's talk economics ;)

You have 10,000 pages and Yahoo crawl all of them for free. Not only that, it would even come back to refresh as often as you refresh your pages.

In return Yahoo gives back 1% or even less in free traffic each month.

If you are happy with that kind of arrangement then good for you.

But that won't work with me. My minimum exchange ratio with any SE is at 10% other than that I'll throttle its access. I'm not paying for bandwidth just to provide Yahoo free content and fillers.

Nor would I pay for 15c or 30c for questionable traffic.


9:01 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Will the free inluded URL be treated the same as the paying URL?

I would say yes, going by how Alta Vista ranks paying and none paying sites. plus with Ask Jeeves going the same route I think Yahoo must stay competitive.


12:15 am on Mar 4, 2004 (gmt 0)

10+ Year Member

Tim - You still have not responded to one of the most presing issues. I brought it up earlier in this thread.

It appears that signing up for PI with Ink and now Yahoo puts your site in danger of getting hit with the "vanishing PI homepage" ban. There are a few different names for the same problem. But several webmasters have reported that signing up for PI with you will put our sites in danger of some type of automated filter or penalty that can't be removed and never will.

Apparently even the resellers of PFI don't even know about it or how to fix it. Or they know about it and deny it because you guys don't know how to fix the bug.

How in the world could we trust a program that seems to have penalized so many legitimate sites without any explanation or recourse?

I'm personally scared to death of PI. Once you sign up, if you get hit by their filter you are gone for good and nobody seems to know how to fix it. I will not even consider Yahoo PI/PPC until this bug is fixed.

I know that my site doesn't have any spam or hidden text or anything, but it seems that other unspamming, clean authority sites have been hit by this penalty for no apparent reason or explanation.


1:23 am on Mar 4, 2004 (gmt 0)

10+ Year Member

I can afford CPC at some level (say, Google at $.05) and have spent $1000's on it already this year. I can do fine with PFI (INK, whether Yahoo is included or not). There is no possible way I can afford $.15 to $.30 per click, period, so my $1,000's stay with Google.


2:39 am on Mar 4, 2004 (gmt 0)

10+ Year Member

figment88 - regarding your Yahoo scenarios in message 327:

1) keywords not competitve. No (or little) Overture, no need to pay PFI, simple optimization should bring you to the top.

2) keywords moderately competitve. Overture listings will take almost all of the searches. Few people will scroll down to regular SERPs. No need to pay PFI.

3) keywords highly competitve. Overture listings take most of the searches, but still in absolute numbers a lot of people venture to the regular SERPs. If the words are so competitive, the margins should be high and PFI should be a bargain.

My understanding is that Yahoo will only show 3 - 4 Overture listings before normal web results. Tim has stated that SiteMatch listings will be mixed with free listings and not given a relevancy boost. If that's the case I think your scenarios 2 & 3 won't apply.

Scenarios 2 & 3 are more like the situation you could get in MSN. i.e. when if there were a lot of LookSmart results Inktomi listings were pushed back so no matter how well a site was optimised and ranking in Inktomi you still didn't make the top 30 results.


3:35 am on Mar 4, 2004 (gmt 0)

10+ Year Member

What concerns me with the new Yahoo! PFI system is a lack of geotargetting. With the current Inktomi PFI, if someone outside my intended area clicks on my page in the SERPs, I don't care - I pay no extra for that. It's just a free click. I currently receive about 15% of clicks from countries I don't do business in.

Under this new system, that would make a $.30 click have an effective cost of ~$.35.



3:40 am on Mar 4, 2004 (gmt 0)

10+ Year Member

...there is an online demo of the Site Match interface available...

For some reason, any account management center that uses the phrase "Fund Depletion" just sends a shiver up my spine.



3:58 am on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

For some reason, any account management center that uses the phrase "Fund Depletion" just sends a shiver up my spine.

lol - allthough they seem to have made topping up again quite straightforward. ;)


4:07 am on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member caveman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

seem to have made topping up again quite straightforward. ;)

Yes, they seem to have made *that* service free. ;)


2:15 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Ok just got back from Pubcon and SES, So a little jet lagged, I was going to go to bed until I saw this thread and OMG what are you people bashing on about, the system is so new no one can give any real results yet, It's just Yahoo Bashing time I guess....

Fact, Yahoo have increase the amount of spidering on all our sites
Fact, Yahoo are a business just like mine and yours.
Fact, when Yahoo switched to Yahink we where in the main all happy.
Fact, Yahoo are pushing more traffic to us then ever before.

Lets give the guys at Yahoo a chance to prove what they are doing will compete with google, instead of just bashing them.



2:29 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member caveman is a WebmasterWorld Top Contributor of All Time 10+ Year Member


OK, now I feel badly about my useless quip in #345.

That said, there are a lot of legitimate questions that do need to be asked, and answered, as webmasters finalize short term planning for the next several months.

Fortunately, Tim is stepping up and handling those, in finest tradition of WW. :-)


2:37 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Caveman yer Tim's a great guy, but if I had to take all that BS, I would go on holiday and not Post for a month :) and let us just pull our hair out.



3:08 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


I think the issue here is not Tim, though he's being somewhat targeted - it's the Y program. But he's use to it - he was sitting in the same seat as an employee of Inktomi a few years back when they took this slippery slope.

For most trying to make a living on the net, this just isn't affordable.

My sites are only worth a few cents per visitor, but they get thousands and thousands of them. So I for one will avoid Yahoo.

For those few that can make a living after Yahoo's cut, I say have at it...



3:23 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

stcrim i agree the model isn't for everyone, but that does not mean in time it won't be.

all I want see is a fair deal for all, and i think that Y! do too, at least thats the feeling i got from tim.



3:42 pm on Mar 4, 2004 (gmt 0)

The issue surely isn't whether they crawl the whole web for free: it's how they index it. Will they stilt their results in favor of the payees in any shape or form?

I'd like to hear a guarantee on that from Tim or Yahoo.

If payment doesn't affect ranking, there is no issue. If it does, at all, there is a serious issue. Not only for all you guys who are trying to make living, but for Yahoo itself, which may very qickly find itself in the same position MSN was in a while back with a diminishing market. Who wants to search on an engine which is dominated by payment rather than pure relevance? Quality non-commercial sites (like ours) would be buried.

Also, wasn't there something from the government about separating pure returns from those influenced by money?

I'd like to see a clear definitive statement on this from Yahoo. If there is nothing to hide, why not clear the air and say so?


3:51 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

Ok, people read the post by Tim that started this thread:

Participation in Yahoo!'s paid inclusion program does not guarantee rank in the search results. URLs are ranked in search results in the same way as all other URLs from the free crawl.


3:56 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

People - the most important thing people are wondering is whether or not PFI influences the ranks and relevancy of search restuls. In order to answer this question, it requires an understanding of how PFI really works.

I was involved with the creation of the first PFI programs, so i know the technical aspects that others may not be aware of and would like to quickly explain the most important aspects.

There are 2 types of PFI:

1. single URL submission
2. XML data feed submission

If done properly and honestly, the single URL submission would not affect the relevancy of the search results since thespider is looking at the actual page that exists and isa pplying the normal algorithm on it and letting that page rank where it may. (I am still skeptical, however that engines do not give these a boost).

The XML data feeds are entirely different. Rather than submitting real, actual web pages, SEM resellers simply take keywords and auto generate massive amounts of "false" pages around those words. These pages do really exist online. The SEM company has the ability to create the Title tag, the description, and the body content.

These pages are fed directly into Yahoo's index. When someone clicks on one of these links, they DO NOT EVER SEE THE FAKE PAGE THAT WAS SUBMITTED. They are redirected to a designated URL within the data feed on the client's actual website - such that no one ever realizes what is really taking place.

While the algorithm is applied on these XML feeds, they are not handled the same as the rest of the web. Here is what i mean:

1. Since the pages are auto generated, they do not have any Link Popularity. This fact could cause the pages to rank very low, so rather than letting that happen, Inktomi applies the same Link Popularity from the REAL site to these fake pages... giving the pages a boost.

2. Since these company feeds can so easily be changed all at once by the SEM reseller, they can very easily tweak their pages in mass until their pages are ranking higher and higher. Since the feed is refreshed so often, they just keep tweaking all the "false" data in the feeds until they hit.

3. As the engines are getting more intelligent and looking at communities and semantics to determine relevancy, it has become more and more impossible to gauge the TRUE relevancy of an auto generated page since it DOES NOT REALLY EXIST IN ANY COMMUNITY.

The recent decision by Ask Jeeves to drop its XML feed PFI program, but keep their Single URL program backs all these points up. In fact, Ask Jeeves explains why the XML feed pages were "VIRTUALLY IMPOSSIBLE" to use but not treat differently than "REAL" pages.

Read what they say here in this press release:


"Concern about how paid inclusion data affected its search engine's relevance led Ask Jeeves to drop its Index Express XML feed program. The company also says the initiative wasn't paying off as expected.

"When I say it impacted relevance," said Jim Lanzone, vice president of product management at Ask Jeeves, "I mean both good and bad. Sometimes [something ranked] better than it should have ranked, sometimes worse."

The company introduced its paid inclusion services in June 2002, but says it's since come to the conclusion it's virtually impossible to merge structured data, such as that in XML feeds, with the unstructured data that comes from crawling the Web.

"It is difficult for the algorithm to understand types of information it's not used to. When we put this information in, the results sometimes showed up in the wrong place," said Paul Gardi, SVP of operations and strategic planning at Ask Jeeves."

I summary, i believe that Yahoo will soon be forced to follow the lead of Ask Jeeves and do away with their XML PFI program. The only reason they have gotten away with it for this long is because people do not understand how the technology is really working and causing the results to be affected.


4:21 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Yahoo has not been very willing to clear up a lot of the issues surrounding their new program. That's too bad, this could be a lot eaiser on everyone if they would.



4:31 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I think that they are doing the best they can at the mo, Tim's the voice at webmasterworld and he is also at SES anwsering questions till the early hours of the morning has he was doing at pubcon!



4:50 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

Hi All,

Tim posted this morning in another thread the new URL for the YAHOO FREE SUBMISSION.


You do have to log into Yahoo first. I have tried it and it seems to be working.


5:14 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Nice post TrafficL.


5:53 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

What I find quite interesting is how the charts do not lie.

Taking a look at the AskJeeves chart (NASDAQ:ASKJ) and its monumental move this morning, you can see with clear direction that this sector is consolidating.

I would have though this paid inclusion thing and pay per click revenue model would have caused a stir but not at this magnitude, it is like 1997 all over again.

YHOO makes it's PFI move while ASKJ takes theirs away. Google sits tight thinking the YHOO move in PFI is a bad test program.

Now you have the Yahoo inclusion program readying the use of Viewpoints (NASDAQ:VWPT) tool-bar for better results and a better user exp. All these companies have huge potential; this is shown in the way they are bahaving for their investors.

Is it smart what YHOO is doing? I really do feel that it is not but this is just my opinion, they really need to put out some press to clear this all up, I think they want to see all of our reactions first thus the delay of clarification. It's like serious free feedback from the webmasters community.

You have to wonder which guy will produce a failure in this ground-breaking search environment that mimics the fall of the Netscape browser.



5:57 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Yup. Should have gotten some ASKJ, when i could have.


6:10 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

An interesting thing, although probably not important is that apart from the fact that the foreign Yahoos are still using Google's Database,

SO IS regular Yahoo.com but it depends on how you enter in your search term. For example:

Instead of putting a space between your words as a search on Yahoo, add in a "+" operator between your words, and the SERP will be identical to a search done on Google for the same words minus the operators.

Also, I know without question that it's very very current.


6:23 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

I also find that the misunderstanding is not such a misunderstanding it is a matter of opinion as quoted below.

It depends who's side you are on, Is Yahoo right or is Askjeeves wrong, my vote is with Askjeeves and not Yahoo as I think a good user exp is the best way to tweak a search engine.

It is very clear to me (In my opinion) that Yahoo is using the broad scale approach and trying to earn more revenue by trying to persuade users to get use to pay for inclusion/pay per click combined while Askjeeves thinks a happy surfer is from a better engine algorhtym. I trust Akjeeves more.

=================== in re below ====================


Ask Jeeves brought in $2.1 million from paid inclusion in 2003, a small sum compared to the $77 million it generated from paid placement. The company carries Google AdWords listings on its sites. "Even if we had all those queries, I don't think it's worth doing," said Lanzone. "I think there are much better ways.... There are probably better mousetraps in dealing with structured data."

Lanzone also noted that, given the other drawbacks, it was easier to drop the program because of the controversy that surrounds XML paid inclusion.

"I could see why people would be suspicious of it," he said. "They would assume that where there's smoke there's fire. I don't want our integrity called into question."

Ask Jeeves will continue to offer the Site Submit program, under which smaller advertisers pay a flat fee to submit their URLs to be crawled.


6:55 pm on Mar 4, 2004 (gmt 0)

10+ Year Member

I think the new Site Match program is going to get website owners to seriously optimize their site's pages only for the MOST RELEVANT keyword phrases, otherwise, you'll get wasted click throughs that doesn't really deliver to users their intended search. It will also push website owners to make their site look and function the best way possible in order to convert that click through to a sale.


2:58 am on Mar 5, 2004 (gmt 0)

10+ Year Member

Ask Jeeves Explains Why The Dropped XML Feed Inclusions
Gary Price prints Ask's reasoning for dropping XML inclusions:

1. Relevance. This is the main reason why we are eliminating Index Express. After considerable testing, we found that this program impacted the relevance of our search results. The reason, we believe, is that Index Express introduces structured (inorganic) data into an otherwise unstructured (organic) Web environment and the inconsistent data feeds are difficult to rank. Since our top priority remains providing users with the most relevant search results on the Web, and we have worked hard to achieve world-class search results over the past two years, we made the decision to eliminate it.

2. Revenue. We determined that Paid Inclusion as it exists today is not the most efficient monetization vehicle for a search engine and dropping Index Express will not have a material impact on revenue for the company. Additionally, we know that user experience drives searches, and increasing search volume is ultimately the best monetization strategy.

3. There must be a better way. Ask Jeeves supports some of the fundamental premises that paid inclusion is based on, notably the need to acquire deep-web content, and the right of search engines to share in the value they create for businesses. However, we have concluded that our Index Express program is not the right model to solve either of these issues.

It is also important to note that while Ask Jeeves is eliminating our Index Express program, we are keeping our Site Submit program in place. Site Submit allows web site owners to pay a fee, per URL submitted, for immediate inclusion of their site into the Ask Jeeves search engine, if the site passes inspection (sites are rejected for spam, porn, etc). Some customers also choose to have their sites refreshed on a more frequent basis. These are actual, organic Web sites who are simply paying to shorten the amount of time it takes for our engine to crawl their sites. We believe this program will continue to be useful to both search engines, users, and site owners. We also believe the model for this program avoids the pitfalls of Index Express, which is why this program has not historically been a hot button issue for those inside or outside the industry.

This 423 message thread spans 15 pages: 423

Featured Threads

Hot Threads This Week

Hot Threads This Month