Yahoo Free Crawl - Pages in the Index, Not in SERP's, Not Penalized

Most interesting

[caveman]

BACKGROUND
A key question we've had about the new Y is: Just how good, meaning how thorough, will the Y free crawl really be? I believe this is at the center of a lot of other questions related to the SiteMatch program, e.g.: Should small to medium sized sites plan on participating in SiteMatch because their pages will not be shown otherwise, or will their pages really be included?

So we had a good look at how our main e-commerce site is currently doing in current (new) Y!.

The site has a couple thousand pages. All of the pages are very easy to find (lots of backlinks) and easy to crawl (no funny characters in the URL strings, no dynamic pages, etc). Nearly all the pages are doing well in G**gle and the other SE's.

About 20 of the pages in this site were submitted to INK PFI over a year ago. These pages are showing well in the new Y SERP's, as expected. A handful of the site's other pages are also showing well in the new Y SERP's.

BUT, nearly 70% of the pages are *not* showing up on searches where I'd expect to see them. For example, our blue widgets page is doing great for a search on 'blue widgets' but our red widgets page is nowhere on a search on 'red widgets' and our green widgets page is nowhere on a search for 'green widgets'.

I have not found this surprising because this reflects my view of the old INK index - *very* spotty.

I assumed that this meant that even though Y Slurp has crawled *all* of the pages in the site over the last couple months, nowhere near all of those pages actually made it into the database, for whatever reason.

Wrong.

WHAT WE FOUND
We realized last night that every single page in the site, except one, is in the new Y! db. We know this because they can be found by typing in the URL strings, which we had never bothered to check before.

I don't think that these pages are being penalized in any way, since they are similar if not identical to the pages that are doing well, they just cover different items/topics.

Rather, it's as if they are sitting in a waiting room, to be formally included with the next update.

Very odd, since Tim had suggested in his post about the Y inclusion program that the Y index was being updated regularly, rather like the G index:

The primary means of generating our index is via our free crawl, using our new Yahoo! Slurp crawler. Yahoo! Slurp discovers pages by following links on the web. We update our index with a daily crawl to gather newly created and fast-changing URLs, as well as our main crawl which updates our index incrementally twice per week.

What seems to be happening is that the more recently acquired pages are now in the index, but are not subject to appearing on standard searches yet. Either that, or 70% of our pages are being individually penalized, but I'm almost certain that is not the case.

Clearly I'm no algo/index expert. Maybe someone else can shed light on the mechanics of this. Those of you who think that your pages have not been indexed yet may want to see if they are there on URL searches.

[Abigail]

I agree completely and I also believe that come April 15 we will see the "new" when they drop the INK PFI - I think the INK will be cut entirely that day and the new index will come flooding in - I hope, I hope...

[textex]

Caveman,
I too noticed that for many of my new sitesm pages are in the index but are not being returned in SERPS.

I don't think its a penalty. I am hoping my like a fresh thing. Like you said, pages in holding area.

[caveman]

One thing I find interesting about this is that if G gets hold of a page and it's well contructed/optimzed, it'll start showing in the SERP's right away. Not so with Y, apparently. Perhaps that will change after the trial period ends...

[helenp]

I do think that if you search for your domain name, and when find pages in db, that db is the old google one, or maybee itīs atw or altavistas db, though I am in db, not in serps and didnīt have any pages in Inktomi before either, and doing very well in google.

[caveman]

You may be right about the listings being old G data...sort of looks like it, though not identical to what G displays, it's close. I guess that might mean that this is *not* likely to stay? hard to know...

<added> Checked a little more. Definitely an old G cached page. Not a great sign, unless it's place holder of some sort, which I doubt. Maybe they're just defaulting to the old G db when they've got nothing to show. Suggests that Y doesn't have it indexed anyway...oh well. </added>

[edited by: caveman at 8:14 pm (utc) on Mar. 5, 2004]

[soapystar]

right. someone else mentioned if they do a site:domain.com search it returns in the exact same order as google. I too have 1500 pages indexed and none being served. when i compare how google and yahoo serve the site:domain the pages are listed in the same order. So seeming to confirm that this is the old google part of the index and yahoo is not serving those pages.

[flobaby]

I found that if I do site:www.domain.com or site:domain.com I get Yahoo results, if I do site:http://www.domain.com I get Google results.

[helenp]

my results:
domain = nothing
domain.com = my index page, and lots of backlinks
www.domain.com = same as domain.com
[domain.com...] = index page, and less backlinks..........this is really strange

[Abigail]

My daughters site is in Google - when I do a www.hersite.com search in Yahoo it doesn't come up at all, she is well rooted in INK and showing in the serps, lots of traffic on Yahoo right at the moment, BUT not showing in the Yahoo index yet. This tells me that Yahoo is still using INK results, and that for some reason they have not picked up her site into new Yahoo index, she is in Google - she has submitted to Yahoo. And yes there are a lot of sites that link to her.

[flobaby]

Helen, just checking...
You did put "site:" before all those (minus the quotes), right?

Nevermind, I see you were just doing a regular search.

[edited by: flobaby at 8:40 pm (utc) on Mar. 5, 2004]

[BobHighland]

I have one page that I have had in Inktomi PFI and it is showing up in the SERPS pretty well under my keywords. This site is also in the Yahoo Directory. However, the directory listing description doesn't show. What is listed is my page title and meta description.

However, if I do a search for my url, the result that comes back is my Yahoo Directory listing. It appears this site is listed twice. The only one with decent rankings in the SERPS is the INK PFI page.

I'm still trying to figure out what all of this means.

[helenp]

Helen, just checking...
You did put "site:" before all those (minus the quotes), right?

?

I wrote , the same way I written.........no quotes

[flobaby]

There's a big difference in results between

site:www.domain.com
and
www.domain.com

If you want to see what pages you have indexed, version 1 is the way to go.

[helenp]

site:domain.com

a lot of pages, and also some files .swf a took away some weeks ago.

For me thats. google

[flobaby]

Funny then how our results differ. Google crawls my site so often it only has new stuff, Yahoo for me is the one that hangs on to the old. And when I do "site:http://www.domain.com", it's exactly the same set of results (and in the same order) as in Google.

Weird, this.

[helenp]

google as well crawls my site every day.
These .swf files arenīt anymore in google searching for Find web pages that contain the term ........

but some are in yahoo index

[Abigail]

site:www.mysite.com - only the INK PFI listings show

www.mysite.com - 1000 pages - also specifying which are the "directory" listings that are in Yahoo

Google packs 1560 of my pages in their index

[helenp]

Oh my good.
They must been mixed up their index.

I am no were to find in Intomi pure search site:www.domain.com, nore just www.domain.com

Seems to be a mix of several index, google-Inktomi mix....
but sites with "penaltys good know why" arenīt shown up in serps.....:)

[Abigail]

Just got to hang in there, I guess and see what happens.