homepage Welcome to WebmasterWorld Guest from 54.226.0.225
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

This 423 message thread spans 15 pages: < < 423 ( 1 2 [3] 4 5 6 7 8 9 10 11 ... 15 > >     
Yahoo Inclusion Program
Trying to clear up some misunderstanding in this forum
Tim




msg:822988
 10:02 pm on Feb 27, 2004 (gmt 0)

The primary means of generating our index is via our free crawl, using our new Yahoo! Slurp crawler. Yahoo! Slurp discovers pages by following links on the web. We update our index with a daily crawl to gather newly created and fast-changing URLs, as well as our main crawl which updates our index incrementally twice per week.

We supplement this free crawl with the paid inclusion program, which allows us to add dynamic database content and other content that we could not otherwise discover and crawl. Currently less than 1% of our index is content that is included via our inclusion programs.

Our paid inclusion programs provide a superior way for sites to interact with us in a clearly structured fashion. These sites get the following
benefits:
1) Control over which URLs they include in the index (subject to content
guidelines)
2) Frequent refresh
3) Clearly defined way to interact with us (for example: clear feed
specifications)
4) Quality review and consultation (for feed customers)
5) Detailed reporting to track and optimize performance
6) Customer service

Participation in Yahoo!'s paid inclusion program does not guarantee rank in the search results. URLs are ranked in search results in the same way as all other URLs from the free crawl. The key drivers of ranking are keyword-specific relevance and site quality, as assessed by our regular search relevance algorithms with input from the quality review process.

(Note that Inktomi Index Connect clients are receiving Yahoo! Search traffic going forward, and Inktomi Search Submit customers are receiving a free trial of Yahoo! traffic until 4/15/04.)

We will also be launching free URL submission during the week of 3/1. This service allows content providers to suggest that the crawler should visit a site during the main crawl cycle. This service is different than paid inclusion in two ways:

1) There is no guarantee that we will crawl and include the URL in the index.
2) As this site potentially could be added to the main crawl, Yahoo! Slurp will follow links from that URL.

This service will reside on the Yahoo.com site and be accessible via a Yahoo! login. This service has been available for the AltaVista and Alltheweb Indices previously and the tradition will continue with Yahoo! Search.

 

Tim




msg:823048
 12:36 am on Feb 29, 2004 (gmt 0)

We are looking into having a feature in the new program where we will rereview your site when you enter the program and give you a chance to get off the blacklist. I will let you know if and when this feature is available.

jonrichd




msg:823049
 12:40 am on Feb 29, 2004 (gmt 0)

Just FYI -- A site I have had been crawled several times by INK and never made it into their DB, except for a few pages. In the new Yahoo index, all the crawled pages were included.

Based on a conversation with Tim, he said that these pages would continue to be in the index, subject to spam guidelines, etc.

So for those of you who have waited forever for inclusion, if you are in Yahoo now, you should remain so in the future.

As far as PFI goes, Tim specifically said that PFI people are getting what they paid for -- inclusion in the then-current INK network. The Yahoo inclusion is a bonus.

Right or not, nowhere did I see in PositionTech that Yahoo was included.

aly4848




msg:823050
 12:57 am on Feb 29, 2004 (gmt 0)

Hi Tim
"2. Customers who have already paid for the current Inktomi program will continue to get all the benefits they have paid for - distribution across the pre-Yahoo! Inktomi network. As an additional bonus, they will get 60 days of free traffic on the new Yahoo! distribution."

Any idea what will happen to UK customers?
Have we been forgoten?
"60 days of free traffic" that finishes on 14/4 would mean INK results should be included by now but uk.search.yahoo.com is still showing google results.

Help appreciated.

Tim




msg:823051
 1:05 am on Feb 29, 2004 (gmt 0)

Aly I will pass this feedback on.
Thanks, Tim

Abigail




msg:823052
 2:39 am on Feb 29, 2004 (gmt 0)

Hi Tim, As some others here, I can see all of my pages in a list of results when I search specific to mysite.com and I have a number of pages in the Yahoo Directory, but I have only yet seen the PFI INK pages in any search results - even searching specific for the name and title of individual pages - is it just "Patience please" or does this indicate a problem for inclusion?

stcrim




msg:823053
 2:58 am on Feb 29, 2004 (gmt 0)

Tim

When you say the Inktomi Network, MSN is basicly the only thing of value and they are spidering and getting ready to take on Google and Yahoo for the sport of it.

Why would anyone want to pay for that knowing it's going to end soon?

And, isn't yahoo going to face the same "very poor" search results Inktomi offered by delivering paid results?

And, if you can get in for free if you wait - don't you think most people will wait?

In all fairness to the searcher, will they be made aware the results that are viewing are paid?

-s-

hurlimann




msg:823054
 3:25 am on Feb 29, 2004 (gmt 0)

And so you should Tim without charge or need to join the inclusion program. Ink banned our site after a competitor used free web space to put some blatent doorways pointing to us and then complained to all the SE's. Ink was the only SE to fall for it and ban us.

Worse still our PFI urls went to the back of the results. A simple check would have revealed we are a top 3 market leader, the competitor is not and has duplicate sites and the spoof domains used resolve to blocks of free ISP DNS they used.

Do Yahoo now want us to pay them to correct this?

cbpayne




msg:823055
 3:26 am on Feb 29, 2004 (gmt 0)

In all fairness to the searcher, will they be made aware the results that are viewing are paid?

Who said the new Yahoo PFI will get a site ranking higher? Why would the searcher need to be notified? Tim spelt out clearly in message 1 what the paid service involves...nothing there about being ranked higher or listed seperatly.

From what I understand, if there are good links to the site, you will be in the index (Yahoo's webamster guidelines say that) and I assume that most of us will not need the Yahoo paid service. (same as the Ink paid service - nothing to do with rankings, just a gaurantee you get crawled every 48hrs - if you alrady rank well at Ink, why bother paying?)

stcrim




msg:823056
 3:26 am on Feb 29, 2004 (gmt 0)

Also Tim,

There is a great deal of concern for those in the Google AdSense program that Yahoo will treat those pages as affiliate pages and drop them from their free and paid database...

Can you address that situation...

-s-

TrafficL




msg:823057
 3:50 am on Feb 29, 2004 (gmt 0)


My 2 cents:

At the end of the day, the fact remains that Yahoo has decided to continue sacrificing the quality of their index in order to make a quick buck.

BOTTOM LINE: PFI creates an inherent incentive to NOT keep the index as fresh and complete as possible since doing so would mean no one would ever pay to be included!

The only way they can create an incentive for people to pay is by claiming that their own engine is not good enough to index the whole web on its own (like Google does), so the only way to ensure that you'll be in is if you pay.

It's great to be hearing Tim and Yahoo reps starting to talk freely about increasing their efforts to spider the ENTIRE web. Up to now, they never talked much about aggresive "free crawling" because they new it would make people feel less of a need to pay. They're spin appears to be that their new PFI program will only consist of content that they "could not find on their own".

Why can't they find this content? If they spent all the energy they're putting into this PFI program towards making their spider and index better, they would be able to find all this so called "hidden" content and refresh their index faster... which will create a much better user experience in the long run.

Dont get me wrong. I am actually a big fan of Yahoo. I'm very impressed with their new engine and am glad they've created some genuine competition in the space. It will only help the industry and the users in the long run. I'm fully confident that in the long run, as users and the general public become more aware, Yahoo will dump their PFI program and will choose to focus rather on building the best possible search engine.

Nacho




msg:823058
 6:30 am on Feb 29, 2004 (gmt 0)

Thank you Tim for sharing Yahoo! efforts to win back the lead in the search world. The team is doing a great job.

Very intresting point of view TrafficL. That does make us all think about how search engines can be more lucrative. However, there is still a need for many sites that have URLs longer than the Chinese Wall and . . . YES, there are still many webdesigners out there that love to do pure FLASH sites and don't understand SEO (and perhaps don't care) because ART is more important than what "view source code" is and how it is viewed by machines (spiders). Therefore, a PI program is a must need for many website owners/designers/etc and a great concept to offer by Yahoo! Perhaps it's things like this that will let them win back the lead.

On a different note:

"2. Customers who have already paid for the current Inktomi program will continue to get all the benefits they have paid for - distribution across the pre-Yahoo! Inktomi network. As an additional bonus, they will get 60 days of free traffic on the new Yahoo! distribution."

I just hope reselling partners (such as PositionTech) will extend the "60 days of free traffic" to its clients . . . or is it Pay Day for them?

EarWig




msg:823059
 12:17 pm on Feb 29, 2004 (gmt 0)

Tim

Your support, feedback and answers to specific questions should be commended. Thanks to Y also for allowing your indulgence in WW

Regards
EW

Tim




msg:823060
 12:45 pm on Feb 29, 2004 (gmt 0)

TrafficL
We look at PI as allowing Yahoo Search to discover additional content to what we would otherwise find by crawling the web. This should logically improve our comprehensiveness and freshness.

mikeD




msg:823061
 1:38 pm on Feb 29, 2004 (gmt 0)

If you are currently in the index and all looks well I would continue creating more great content that we can crawl for free. I think the only change you will see in the future is greater comprehensiveness and freshness.

These kind of comments fill me with confidence, especially,

is greater comprehensiveness and freshness

I do feel Yahoo can catch up and genuinely challenge Goolge. And it may not take too long. What they need to do is make sure the cream rises to the top a bit more.

I have been quite a critic of Kelkoo's top ranking in Google, but I have to say in the commercial serps I think Yahoo could do with sites like this ranking better. Because what is beating Kelkoo is far worse in terms of quality and relevancy.

stcrim




msg:823062
 3:08 pm on Feb 29, 2004 (gmt 0)

>>>if there are good links to the site, you will be in the index<<<

Than who in their right mind would pay for it?

-s-

TrafficL




msg:823063
 3:08 pm on Feb 29, 2004 (gmt 0)

Tim:

Can you give us an example or two of content that your search engine spider currently is not able to find on its own?

... and assuming there ARE examples of that type of content, why not just improve your spider such that next time it CAN find that content on its own?

In my opinion, relying on PFI to find this content is not the best way to ensure that this "hidden" content finds its way into your index. The only true way is to build a spider that can find it on its own. This is the route Google seems to be taking.

Also, if your spider does improve such that it can find this content on its own, would you discontinue this PFI program, or would you purposefully not let your spider include it since you would than be turning down payment for its inclusion?

(Tim - i appreciate that Yahoo has given us the opportunity to interact with you and provide feedback like this. You've been a great source of feedback for us all here at WebmasterWorld. I am NOT critical of Yahoo... i think they're great, I am just a firm believer that PFI will hurt Yahoo's reputation in the long run. I understand that other side of this argument and I think this is a healthy debate.)

Chndru




msg:823064
 3:26 pm on Feb 29, 2004 (gmt 0)

From msg#1
The key drivers of ranking are keyword-specific relevance and site quality, as assessed by our regular search relevance algorithms with input from the quality review process.

quality review process?

To me, it looks like a big bloated yahoo shopping index than a search index.

walkman




msg:823065
 4:08 pm on Feb 29, 2004 (gmt 0)

"quality review process? "
tha sounds weird, but I don't think they can review that many sites. Also Tim said that PFI pages don't get a boost so I don't know.

flobaby




msg:823066
 4:45 pm on Feb 29, 2004 (gmt 0)

PFI pages don't get a boost

This is true. I have a redirected page pointing to a PFI page and because of the way Ink works, both URLs are listed seperately in the database, though it's the same page. The PFI page is 13 places lower than the redirected page.

walkman




msg:823067
 5:07 pm on Feb 29, 2004 (gmt 0)

"both URLs are listed seperately in the database"
It is my BELIEF that PFI and Google pages (still on Y! I think) are not calculated in the algo, therefore only show up in the end as a backfill.

flobaby




msg:823068
 5:23 pm on Feb 29, 2004 (gmt 0)

Just to clarify, it's an Ink thing, not just Yahoo, because MSN and Hotbot show the same anomaly.

wellzy




msg:823069
 5:51 pm on Feb 29, 2004 (gmt 0)

Tim
If a site has some, but not all, of it's pages in Yahoo will Yahoo! automatically crawl the rest eventually? Or will we have to do PI?

On a more serious note, how come you are not Yahooman? ;)

Scarecrow




msg:823070
 6:25 pm on Feb 29, 2004 (gmt 0)

Slurp apparently won't go deep on noncommercial sites. It did okay for a few months at the end of 2002, but for the last year it has been pathetic.

My site has over 50,000 "eyeball" page views a day (all bots are excluded from this count), and has 120,000 static pages. Google has covered it fairly well now for three years, by including anywhere from 60,000 to 90,000 pages from this nonprofit, tax-exempt site in their index.

For the last year, Slurp has had only 400 pages in their index from this site. What's curious is that the site has no competition, and there is no other way for anyone to find this information on the Web. But Slurp doesn't care. The question in my mind is how committed will Slurp be when it comes to deep-crawling the noncommercial sector? I'm not optimistic.

Right now, I don't need Yahoo. It seems that most noncommercial information seekers are already conditioned to use Google. When Yahoo dropped Google, my page views dropped only about eight percent.

outland88




msg:823071
 6:48 pm on Feb 29, 2004 (gmt 0)

I can't understand if I've been paying Positiontech for years, and still am, why I'm not included in Yahoo and Inktomi right now. To me it looks like the supposedly two differing databases have the same penalty. If its a new database it seems to me the same situation wouldn't be present in both.

As for a promised review of this matter by Yahoo. PT suggested I contact about 15 search engines and request removal of my listings because Inktomi viewed this as duplicate content. Am I going to be listening to the same craziness from Yahoo hourly employees, with a review process. Plus suggestions to alter my site that has never been penalized with any other engine.

Quite frankly I'm seeing bullentin board sites in Yahoo with 10,000 pages of spam and I can't get a small site in.

walkman




msg:823072
 7:10 pm on Feb 29, 2004 (gmt 0)

"the supposedly two differing databases have the same penalty"
Looks like Yahoo started with the Ink database and that included the "ban" list. After all they bought Inktomi so they didn't have to re-invent the wheel. This dupe content has to be automated. If you find dupes, you pick the one with most back links. No need for lifetime bans.

dougs




msg:823073
 9:22 pm on Feb 29, 2004 (gmt 0)

We have a client site that we have some pfi pages listed and on Feb 9th all of our traffic from ink etc stopped.

Care to comment Tim on what happened here? Was that reltaed to the new search being rolled on on Yahoo as well as MSN

Doug

steveb




msg:823074
 10:10 pm on Feb 29, 2004 (gmt 0)

"quality review process" = "we sometimes hand remove sites we consider spam"

atadams




msg:823075
 10:19 pm on Feb 29, 2004 (gmt 0)

Customers who have already paid for the current Inktomi program will continue to get all the benefits they have paid for - distribution across the pre-Yahoo! Inktomi network. As an additional bonus, they will get 60 days of free traffic on the new Yahoo! distribution.

Actually, I paid (on Tuesday, no less) to be in Yahoo. Seeing a drop in traffic I clicked on the "what's this" link on a Yahoo SERP and read the following text:

Web Results are the most relevant web pages found by Yahoo! Search Technology in response to your search terms. Web Results may include sites that participate in the Inktomi Index Connect™ or Search Submit™ programs. Sites from both of these programs pay to be included in our search engine index, and for Index Connect, sites also pay to be reviewed for relevance.

So I followed the links and signed up for a year. But that's now what I'm getting.

I think I will call my credit card company tomorrow and dispute the charge.

Tim




msg:823076
 10:37 pm on Feb 29, 2004 (gmt 0)

Dougs- it is hard to diagnose the problem but you can send your domain name to webmasterworldfeedback@yahoo.com and we try to diagnose it.

ATAadams- If you could be more specific I could maybe answer your question. Is your post a question or a commentary?

dougs




msg:823077
 11:05 pm on Feb 29, 2004 (gmt 0)

Cheers Tim.

Doug

lost in space




msg:823078
 11:08 pm on Feb 29, 2004 (gmt 0)

Tim,

I have many quality sites listed in the Yahoo Directory, but none have been included in the new Yahoo Search results. Will the new spider find them via crawling the Yahoo Directories or do I need to submit them. Also is there going to be a limit on free submissions?

Thank You & Yahoo for supporting webmasters!

This 423 message thread spans 15 pages: < < 423 ( 1 2 [3] 4 5 6 7 8 9 10 11 ... 15 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved