homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

This 423 message thread spans 15 pages: < < 423 ( 1 2 3 4 [5] 6 7 8 9 10 11 12 13 ... 15 > >     
Yahoo Inclusion Program
Trying to clear up some misunderstanding in this forum

 10:02 pm on Feb 27, 2004 (gmt 0)

The primary means of generating our index is via our free crawl, using our new Yahoo! Slurp crawler. Yahoo! Slurp discovers pages by following links on the web. We update our index with a daily crawl to gather newly created and fast-changing URLs, as well as our main crawl which updates our index incrementally twice per week.

We supplement this free crawl with the paid inclusion program, which allows us to add dynamic database content and other content that we could not otherwise discover and crawl. Currently less than 1% of our index is content that is included via our inclusion programs.

Our paid inclusion programs provide a superior way for sites to interact with us in a clearly structured fashion. These sites get the following
1) Control over which URLs they include in the index (subject to content
2) Frequent refresh
3) Clearly defined way to interact with us (for example: clear feed
4) Quality review and consultation (for feed customers)
5) Detailed reporting to track and optimize performance
6) Customer service

Participation in Yahoo!'s paid inclusion program does not guarantee rank in the search results. URLs are ranked in search results in the same way as all other URLs from the free crawl. The key drivers of ranking are keyword-specific relevance and site quality, as assessed by our regular search relevance algorithms with input from the quality review process.

(Note that Inktomi Index Connect clients are receiving Yahoo! Search traffic going forward, and Inktomi Search Submit customers are receiving a free trial of Yahoo! traffic until 4/15/04.)

We will also be launching free URL submission during the week of 3/1. This service allows content providers to suggest that the crawler should visit a site during the main crawl cycle. This service is different than paid inclusion in two ways:

1) There is no guarantee that we will crawl and include the URL in the index.
2) As this site potentially could be added to the main crawl, Yahoo! Slurp will follow links from that URL.

This service will reside on the Yahoo.com site and be accessible via a Yahoo! login. This service has been available for the AltaVista and Alltheweb Indices previously and the tradition will continue with Yahoo! Search.



 4:26 pm on Mar 1, 2004 (gmt 0)


My point exactly! And since Yahoo isn't doing PFI for their health, I bet you're gonna see the same thing there.

They are not out to help the lowly webmaster get along in this world - this is a for profit system they are building and all the sugar coating in the universe isn't going to change that...



 4:28 pm on Mar 1, 2004 (gmt 0)

Rumours of Google's imminent demise in this forum are premature. I have a site that ranks very well in Google (again after having disappeared after Florida/Austin etc) and is also well positioned in the new Yahoo index - first page for the same phrase.

The last 40 referrers include Google: 14, MSN: 5, Yahoo: 1.


 4:31 pm on Mar 1, 2004 (gmt 0)


You have probably heard this before as I believe many other site owners are experiencing the same problem as I am experiencing. Before the switch by Yahoo from Google to Yahoo and Inktomi my site was receiving a good ranking in the SERPS in Yahoo, Google and some other search engines. After the switch my site fell to the bottom of the SERPS in Yahoo for no apparent reason. It still ranks well in Google, Alta Vista and some other search engines. I had not overly optimized my site. In my opinion my site has quality content and is linked to by other quality sites. My concern and question is why am I having this problem and what can I do about it?
I have PFI in Yahoo via Net Solutions and Lycos InSite.


 4:40 pm on Mar 1, 2004 (gmt 0)

My point exactly! And since Yahoo isn't doing PFI for their health, I bet you're gonna see the same thing there.

I have the same questions, based purely on INK's current behavior and past history. However, past INK is not present Y! My hope, just bolstered by Tim's comment, is that Y! really strives to be a great SE with it's own proprietary search...while still making the profit it needs for shareholders, etc.

This could be achieved, it seems to me, by:
--Showing *all search results* for a given query, *unbiased by PFI*,
--updating the index on a regular but not too frequent basis (e.g., every 4-8 weeks),
--updating PFI pages every 48 hours.

That way, those who want frequent spidering would need to pay. Those who don't, or can't afford it, would still be in the index, with the same shot at having a presence as the big guys.

Works for me, anyway. :-)


 4:55 pm on Mar 1, 2004 (gmt 0)

Tim, thank you for talking at PubCon, you are one of the reasons I made it out :) I am glad to see Yahoo playing a part of the community by interacting with everyone. Good move and thanks for the notes on here :)


 5:05 pm on Mar 1, 2004 (gmt 0)

caveman, you spelled out what the ideal should be and what most think is fair. Divert from that too much and you get into the realm of sponsored listings masquerading as something else. I'm sure Yahoo knows this and is also aware that Google's attorneys would be happy to point this out to the Feds.


 5:19 pm on Mar 1, 2004 (gmt 0)

does that mean that Yahoo is going to handle this differently than Inktomi?

I can assure you that Yahoo will handle this properly and not affect free pages in the index. The only reason we will delete URLs is if they violate our content policies and are degrading the search experience.

The new program will have many benefits but higher ranking is not one of them.


 5:40 pm on Mar 1, 2004 (gmt 0)

>>>I can assure you that Yahoo will handle this properly and not affect free pages<<<

That remains to be seen.

And to cost - if you get almost nothing (MSN) for $25 per page per year - what's it gonna cost for Yahoo PFI who can deliver real traffic?



 5:56 pm on Mar 1, 2004 (gmt 0)

I think it would be very wise for Tim and all of Yahoo to listen carefully to the common sense advice StCrim and Caveman are giving here regarding PFI.

I'm very encouraged by Tim's comments that in the NEW Yahoo search, if you pay to have 1 page included, they will still spider the rest of your site for free. However, since I fully understand how Ink's current PFi works, I admit that I am skeptical that this will happen since it will deincentivize people from paying. We shall see if this statement holds true.

I am also encouraged by Tim's remarks that in the NEW Yahoo search, money will NOT influence rankings since i know that with Inktomi, their Indext Connect XML feeds DO allow SEO firms to manipulate ranks in a tricky way using cloaked doorway pages. (Tim - feel free to clairfy this if its not true anymore.)

It does appear that Yahoo is listening and is starting to "come out of the closet" with how PFI really works and is trying to be more up front as to how money influences their results. Lets hope this continues.


 6:43 pm on Mar 1, 2004 (gmt 0)

I went from penalized in Inktomi/Yahoo yesterday to completely unlisted today. Well atleast paying Positiontech got me something! Perhaps if I pay Yahoo or Ink again I can go back to being just penalized.


 6:53 pm on Mar 1, 2004 (gmt 0)

I really enjoyed Tim's presentations at the Pub Conference. I was impressed with his overall awareness of the issues that many webmasters have with PI and the Inktomi PI problems. I was also encouraged by his overall attitude toward the webmaster community in general and his willingness to address these issues.

Many of you know that I have been dealing with significant problems since I signed up for PFI.
I won't go into the deep details in this thread.

Instead, I wanted to share my overall thoughts regarding PFI in general (on a philosophical level). I don't want to touch the whole "trusted feed” issue at all (another discussion completely). I’ll just talk about search submit (or the submission of one page at a time for inclusion in a search engines database).

Even though I have had huge problems with PI, I still think PI can be an excelent model if it is done right. In theory, it can be beneficial for all parties involved;

-As a website owner, you get your pages refreshed regularly and they are "guaranteed" in the index. You also get the peace of mind (in theory) of knowing that they will always be in there.
-The searchers get up-to-date relevant data in the SERPS.
-The engine gets $.

It's a different model than Google, but it can definitely work…if it functions how it is designed to work. It will only work (IMHO) and if they mix the PI results with LOTS of freely-crawled pages. According to Tim, I think he said that less than 1% of the Yahoo database is currently PI... so perhaps they feel the same way. It remains to be seen if they are actually motivated to get out and crawl the web. Time will tell.

For me PI didn’t work how it is supposed to. Actually, it worked the exact oposite. My PI page vanished almost immediately after paying for inclusion. It’s cost me hundreds of hours trying to track down what went wrong and LOTS (and I mean LOTS) of $ in lost traffic and sales. Still, in theory, I think it's a good model and it can work if it is executed correctly and if Yahoo puts some customer service into the equation and allows you to know if your page/site has been tripped by some kind of filter or penalty. The lack of customer service/knowledge was one of the most frustrating and unfair things about Inktomi's PI system. We were hit with some kind of automated filter and our PI page (our home page which drove almost all of our traffic) vanished from Inktomi. Unfortunatley, our reselleter swore to me that there was "absolutely no penalty" at all (when in fact there was). It was the most frustrating experince I've ever had with any program/advertising I've ever had in 6 years of marketing on the web.

If Inktomi had worked how it was designed to work, I was ready to submit 40-100 PI pages to the Ink database. Ink would have made considerable $ from me and I would have been happy with my inclusion and rank. I've been holding off on further PI since my first experience was so poor, but if we can get it fixed, we will still use PI for several of our pages. I like the idea of being able to tweak your pages on a daily basis and see the results 48 hours later and also the opportunity to have fresh content in the Yahoo database every day. If something in my industry changes and there is some breaking news, I can change my PI page and get to the top of the serps for those terms as consumers search for those topics at Yahoo.

Anyway, I thought that some might be interested in hearing my honest thoughts on PI....so there you go. I guess my point is that just because I had a bad experience, doesn't mean that the model is flawed. The model can work, but Yahoo's got to figure out how to overcome some hurdles. And again, I'm just talking about "searh submit" not "trusted feed".

It's going to be interesing to see how it plays out, that's for sure!

One thing I took away from the conference was that Yahoo is intent on making this work... and they are listening to the webmaster community to get feedback on how to make it work. That, alone, is really good news. I am definately rooting for Yahoo! Even though I do VERY well, on Google, the diversification is good for the market and good for webmasters' in general.


 7:34 pm on Mar 1, 2004 (gmt 0)

For whatever is worth, to me, the PFI route just shows
  • the lack of faith/skill by yahoo to monetize their organic serps
  • the failure to recognize the value the natural serps bring to their searchers.

    I for one, will never use any SE, in which someone pays to get "included"

  • stcrim

     7:36 pm on Mar 1, 2004 (gmt 0)

    >>>I am also encouraged by Tim's remarks that in the NEW Yahoo search, money will NOT influence rankings <<<

    Without the ability to tweak like the XML feeds none of the larger sites would have the least bit of interest in the product.

    Can you see Amazon submitting all their ever changing pages or EBay - one page at a time through PositionTech.

    Without the ability to tweak their way to the top the big players would never look at Yahoo PFI twice.



     7:49 pm on Mar 1, 2004 (gmt 0)

    Chdndru-- That is a fairly rigid stance. If PI is the avenue into a SE database like Yahoo it seems you have to go along for the sake of your company or your clients. The game is not onesided any longer where the website owner makes the cash at the expense of the search companys bandwidth.
    Free search is going away.
    My clients will accept the pragmatism of paying Yahoo.
    All this makes you wonder why MSN dropped LookSmart doesn't it? Everyone loved to bash it but it seems they had the model along.


     7:53 pm on Mar 1, 2004 (gmt 0)

    Free search is going away.

    All this makes you wonder why MSN dropped LookSmart doesn't it?

    Because every day since this whole deal started, people proclaim the death of free search. The PFI models are the only ones disappearing.


     8:07 pm on Mar 1, 2004 (gmt 0)


    There have been a couple of other threads about your issue. I am having the same problem. Soapystar speculates that it is a duplicate content penalty caused by other sites linking to you using redirects, so that it looks like your content is on their site. This is consistent with what happened to me. Tim asked us to email webmasterworldfeedback@yahoo.com, and says they will look into it. In the mean time, see if you can find out who is duplicating your content, and see if you can get them to stop.


     8:19 pm on Mar 1, 2004 (gmt 0)

    >-As a website owner, you get your pages refreshed regularly and they are "guaranteed" in the index. You also get the peace of mind (in theory) of knowing that they will always be in there.

    What guarantee? This eliminates Yahoo's ability to remove the page due to a penalty. I doubt Y! will paint themselves into this corner.


     9:09 pm on Mar 1, 2004 (gmt 0)

    Hello Webmasterworld, from a long-time lurker, first-time poster. . . thank you all very much for your contributions.

    stcrim has exposed something very important:

    In INK if you had a page in the db and paid for one the freebie would (or could) remain but you would never get a new freebie.

    From personal experience, I believe that this is 100% correct. It is disappointing to think that because I paid for a service, my site is now flagged and is effectively being penalized (by having to pay for all future pages to be included). And it is incredibly frustrating to think that "if I had not paid for PFI, my site would be in the index". Out of principle, I will not do business with a company engaging in such deceptive practices. I hope that Yahoo does not continue this tradition.

    I did see Tim's response about the new Yahoo search:

    >>>Will having a few pages on PFI hurt the chance of getting the rest of that site indexed for free?<<<
    No it will not in Yahoo Search.

    which is encouraging, but I think that this is a topic that deserves continued attention over the coming months. I apologize that my post is somewhat of a repeat, but I wanted to support stcrim's assertion, which has profound implications.



     9:56 pm on Mar 1, 2004 (gmt 0)

    Inktomi sucked. Inktomi is close to dead. Yahoo is not Inktomi.

    Free search has won the battle. It's over. Yahoo PFI sounds to me like its for dynamic pages that you want for sure in the index, and for people who (basically) want to pay for spidering.

    This at least is my hope: that Yahoo will get far better at free crawling and include essentially everything it should, but that you can additionally pay Yahoo for the equivalent of Google Freshbot.


     10:55 pm on Mar 1, 2004 (gmt 0)

    1st time poster as well. Guess hot topics bring people out of the woodwork...

    Tim, you said...

    "We are working on including all the quality sites in the Yahoo Directory. We have many of them but not all. Please be patient. "

    Would you please elaborate on "quality sites in the Yahoo Directory." The $299 aside (which Yahoo previously kept even if site was rejected), I thought sites accepted in Yahoo Directory already went through a stringent and subjective editorial process.

    Thanks for the informative feedback


     11:30 pm on Mar 1, 2004 (gmt 0)

    "I am having the same problem. Soapystar speculates that it is a duplicate content penalty caused by other sites linking to you using redirects, so that it looks like your content is on their site."
    I would agree. This could be your cause for removal. I have spent literally hundreds of hours studying sites and the Ink serps. But it's hard to just put a "blanket" explanation for the "vanishing PI" conundrum. It looks like it could be one (or more) of a few factors.

    1. Other sites linking to you through redirects. In this case Ink thinks that the redirect is actually the original page and thinks that yours is duplicate content spam. Then Ink penalizes your page for what another site has done (this seems especially true if the site linking to you was on the web before your site). This definitely seems to be a problem.

    2. Lots of inbound links from one site in the same sphere as yours. Once again, this can be something as simple as navigation on your own site (if you use sub domains or have a Yahoo store etc.) or lots of inbound links from another site... I guess theoretically, a competotr could put up hidden or inconspicious links to your site from some random junk domain, get your PI page banned. This is theoretical... I have no idea if it would actually work.

    But, these seem to be the two most reported problems. I've seen lots of evidence of both problems. If indeed these are the reasons for the penalties, it's very problematic because they allow outside influences to determine if you get banned.

    I think Ink (and Yahoo is now using Ink's technology) needs to work very hard on the accuracy of their automated filtering system. Any automated "penalties" put on sites, in my opinion, should not be subject to off page factors at all and should be dependant on human intervention at all times. Otherwise, it leaves far too much room for a competitor to screw with results. Any engine that allows outside sites the ability to get another site penalized will have an epidemic on their hands within a short amount of time.

    If they are going to implement automated penalties on PI pages, then they also need to provide an avenue for PI customers to have a Yahoo rep review the penalty and remove unwarranted penalties within a short amount of time (24-48 hours). Especially when they are "guaranteeing" inclusion.

    I'd be willing to pay for a review service like this and I'd pay a lot of money for it too. My site is absolutely clean... I have no problem having any rep from any search engine go through it. We follow all guidelines and we are very careful to abide by rules etc. I was hit as collateral damage in the Ink crusade against spam. I can appreciate Ink/Yahoo's attempt to combat spam, but it's very frustrating when you get tagged for things that you did not do.
    From what Tim alluded to in his presentations at the conference, Yahoo is well aware of these issues and they are looking at ways to solve these problems. Yahoo is progressive and forward thinking. I'm sure that they will get something in place fairly soon to solve this problem and to make the process better.... in the meantime, I'm getting killed.


     12:23 am on Mar 2, 2004 (gmt 0)


    I am having problems as well,
    How could I check if my site are having these redirections problems?

    seing some 301 and 302 in my site, though I don´t have any, but seem to be errors


     12:43 am on Mar 2, 2004 (gmt 0)

    I can see a number of hits in my logs for Yahoo - when I go to find them I see other urled pages, not my url showing on the serps, and when you hit them they go over to my page - why would someone even want to do this?


     12:49 am on Mar 2, 2004 (gmt 0)

    >>There have been a couple of other threads about your issue (msg #:123). I am having the same problem. Soapystar speculates that it is a duplicate content penalty caused by other sites linking to you using redirects, so that it looks like your content is on their site. This is consistent with what happened to me. Tim asked us to email webmasterworldfeedback@yahoo.com, and says they will look into it. In the mean time, see if you can find out who is duplicating your content, and see if you can get them to stop.<<


    I wouldn't think Yahoo would penalize a site because another site is duplicating the site's content. If that were the case one's competiter could do one great harm. I do have a few pages on my site that duplicate a few sentences, but nothing like a duplicate page.


     12:55 am on Mar 2, 2004 (gmt 0)

    has anyone seen this?

    "Yahoo VerticalCrawler FormerWebCrawler"


     1:15 am on Mar 2, 2004 (gmt 0)

    "why would someone even want to do this?"

    It's not uncommon to use a tracking code on a link that runs through a cgi-bin.

    No point in mincing words, Ink has completely bungled this. Yahoo needs to work hard now to fix this major problem. Quite simply they penalize the original site while ranking sites that copy content or link via a redirect. And, quite simply, you can do dramatic harm to competitors by creating such links and/or copying their content.


     1:56 am on Mar 2, 2004 (gmt 0)

    Are we sure about this penalty? I mean, I have literally thousands of sites that link to mine and many, many, many of those do this redirect thing.


     2:39 am on Mar 2, 2004 (gmt 0)

    No... that's the thing. Because it's an automated penalty, it's almost impossible to figure out exactly what tripped it. Many of us are certain that it's off-page factors. I've looked at over a dozen sites that have been hit with the vanishing PI page.

    Interestingly, dupe content (caused by redirects or other factors) seems to be a common factor. I spent somewhere at least 200 hours (actually probably a lot more)on this issue... I've sent dozens of emails to multiple resellers and spent hours on the phone (both on hold and talking to them). I've analized serps for over 9 months straight trying to figure out what in the world happened to our site after paying for PI. I've compared sites. I called Inktomi. I've reviewed tons of sites from around the web from webmasters experiencing the same problem. I've scoured the web in search of common factors and I've found forums across the web reporting the same probelms. I've changed our home page multiple times. I've tried everything that my reseller suggested (including deleting our other highest driving page in Inktomi). I can't tell you how much sleep I've lost over this issue and how much research I have done on it.

    Ink treats some redirects correctly and others it treats the dynamic link as if it were the new content. I can't figure out what causes Ink to incorrectly index some of them and to do fine with others. After all this time... it's still a mystery.

    One thing is certain. We are the leader in our industry and the authority when it comes to sales and traffic. We are keyword.com (the keyword IS our industry). Our absence in Yahoo is like not having Dell show up when you search for "Dell Computers." We have never done anything to spam any engine.

    We are clean.

    We had two Inktomi editors go through our entire site with "a fine-toothed comb" (It's a big site) and they couldn't find anything warranting a spam penalty at all.

    [edited by: kanetrain at 2:53 am (utc) on Mar. 2, 2004]


     2:52 am on Mar 2, 2004 (gmt 0)

    "We had two Inktomi editors go through our entire site with "a fine-toothed comb" (It's a big site) and they couldn't find anything warranting a spam penalty at all."

    and they still didn't remove the penalty? Geeez...


     2:58 am on Mar 2, 2004 (gmt 0)

    Two thoughts?

    First it's very hard to get a penalty from a search engine. The tricks spammers use are also very common in normal web developing. Redirects being one of the most common.

    Yahoo purchased Inktomi to buy up the competition and use the technology. So what makes anyone think Yahoo PFI is going to be any different than INK PFI - after all it's being developed by the same people for exactly the same use.

    All the promises in the world being offered here aren't going to make that any different. Yahoo is doing PFI for profit not to create a better database as offered earlier.

    Forums like WebmasterWorld have turned out to be great marketing tools for search engines. It's a free and easy way to whip up a lot of business in a hurry.



     3:26 am on Mar 2, 2004 (gmt 0)

    "and they still didn't remove the penalty? Geeez..."

    Not yet. But we are very hopeful though. Because the penalty was an automated one (placed on us by the engine not a person) it appears that it is more complex to remove. We are practicing patience to the nth degree. ;)

    Tim spoke at Pubcon about adding an element of customer service to PI that hasn't been there in the past. It was VERY encouraging!

    For anyone who thinks I am bluffing or blowing smoke... a few weeks ago, we hired an very reputable and well known SEO firm to go through our entire site in search of any possible reason for any penalty (and yes, we paid quite a bit for it too). They were baffled and simply confirmed what we already knew... they couldn't find anything. They guessed that the only possible reason would be lots of inbound links from our own Yahoo store (navigational purposes only to get from our Yahoo store to the other content areas of our site).

    In fairness to Ink... perhaps this was some penalty that was levied more than a year ago, before I owned the site. And maybe it just didn't surfaced or affect us until we signed up for Paid Inclusion. I really don't know.

    This 423 message thread spans 15 pages: < < 423 ( 1 2 3 4 [5] 6 7 8 9 10 11 12 13 ... 15 > >
    Global Options:
     top home search open messages active posts  

    Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved