Yahoo and Large Sites

what's the deal?

[mfishy]

The new Yahoo! search seems pretty straightforward in regards to scoring, but what is the deal with getting large sites fully indexed?

Site's have their index page #1 for many terms yet 5 out of 10,000 interior pages are showing. These pages get crawled all the time. Overall, Yahoo search has been surprisingly good but their inability (or unwillingness) to index large sites will not help them. Any thoughts?

[caveman]

Are you convinced it's large sites primarily? We have numerous mid-sized sites with only 5-20 pages showing, despite visits from the bot, good inbound links to index and subpages, etc.

As you say, inability...or unwillingness?

My best hope is that the new index presumably to be unveiled on 4/15 will have a lot of those pages that are being spidered but not displayed right now. I haven't bet any money on that, however.

As a sidebar, we do see the number of pages appearing from those sites slowly going up. But very slowly.

[worker]

Just pay to have them indexed.

; )

You only have to pay once per URL and then only every time there is a click that leads to one of the pages.

[Seattle_SEM]

I've seen the exact same thing - I'm sitting on a 100k page site, launched in December '03, that has *three* pages in Yahoo!, and 20,000 in Google.

;-(

[Scarecrow]

I'm on a 128,000 page site with anywhere from 50,000 to 80,000 in Google. Google's been that way for three years, except for a year ago, when it dipped to 35,000 for one month.

Yahoo has fewer than 400 pages from my site, which came from Inktomi. Ink went from almost almost nothing to suddenly showing 50,000 pages indexed in August 2002, and then dropped all but a few hundred last April. No rhyme or reason. I was doing quite well on MSN for a few months, thanks to Ink. But that's ancient history now.

Mine is a unique nonprofit site with no competition, and about 18,000 Google referrals on a weekday.

The crawling from Slurp has been anywhere from a hundred to a thousand pages per day, for as long as I can remember. It all goes into a black hole somewhere. I wish I could forget about them, but Google is starting to make me nervous lately.

[mfishy]

<<Are you convinced it's large sites primarily?>>

No, it's all sites.

Guess it just hurts more when you are missing 99,999 pages instead of 15. OTOH, it's pretty sad that they can't get a full 20 page site in when the index page is in their directory and crawled everyday...

<<every time there is a click>>

This particular site can not afford $.15-$.30 per click. I would imagine many informational sites are the same way.

[worker]

Maybe Yahoo would index the whole site and not charge per click since the site is informational.

Is there an email address where people can send in requests to have their site reviewed for full free inclusion like in Google?

[caveman]

>>No, it's all sites.

Yep, at least from where I sit.

Their relevance/usefulness is looking pretty good overall vs. G (albeit) a bit spammier ... but Y's index has more holes than swiss cheese...