What triggers Yahoo to fully index a site?

[arran]

I've read quite a few threads recently regarding Yahoo's tendency not to fully index new sites. If we exclude the possiblity of a site being banned, the usual recommendations in these threads are:

- Get more links
- Get deep links
- Wait

Can anyone who has gone from homepage-only to fully indexed share their experiences? For example:

- Has anyone been successful with a small number of quality inbounds and if so, how many of these were deep links?
- Has anyone managed to get a site fully indexed quickly? If so, what linking strategy was used?

I have 2 sites (both ~ 4 months old) which only have 4 and 5 pages indexed by Yahoo respectively. Slurp has deep crawled these sites a couple of times in the past month.

[RichTC]

IMO for what its worth, i think the yahoo index is full with not much room for new content.

Our site was live Oct 2003 we are a substantial sized UK site, We must have 9-10k of links to it, Yahoo visits every day but takes little content to index - i think we have about 1% of our content in the index about 2700 pages.

I think its a case of keep waiting, it moves at snail pace and adds a few pages - some sites i know that have good percentage of pages indexed are in most cases over 5 yrs old. Others i know are in a similar boat.

Meanwhile i guess if you are a US site, that Yahoo will want you to use its premium index service where you pay per click in the SERPs but they index all of your pages for you.

Its amazing they can index youre entire site if your paying!

[Vulcan315]

Our site is over 7 years old, has several hundred pages indexed by Google, hundreds of good inbound links from around the planet, but only one page is indexed by Yahoo.

Dave

[jaffstar]

Can anyone who has gone from homepage-only to fully indexed share their experiences?

Welcome to the world of the yahoo penalty. Nothing seems to get you out of it. I have contacted them half a dozen times, and nothing.

Clean site, unique content, yahoo does not like it as its in a competitive industry.

[Voxman]

When Yahoo relied on Google for the index we had over 100,000 pages indexed (in both Y & G of course..ahh the good ol days) and last year around this time went down to 1,100 in Yahoo. Dam near lost our business because of that. What made it worse was they had pages indexed that were old and out dated from the Inktomi days (our site goes back to June 1998)

Around Feb we went up to 1,800 and then 2,200 pages (our site now has about 140,000 pages of product which is database driven.

In the last few days it's been all over the place but generally up. Showing 6,000 right now but I have seen 12,000 and 14,000 as recently as hours ago. They are all over the place. I'm starting to think they don't have the resources or power to index a very large site. Google has 87,000 of our pages indexed for a comparison. It's not penalty in our case.... they are just using the out dated Inktomi system and slurp sucks to be blunt.

I just hope for the best now as we have learned to get by almost solely on our Google placements. We subtly make them pay by giving Google more of our advertising dollar (I know they don't care but it makes us feel better)

By looking at their index we would have probably would have been better off being an affiliate of ourselves rather than the main content site. Affiliates have completely spammed both indexes (Y & G) and got away with it. Hopefully they will wake up someday and realize what a mess the index has become.

Wow....when I rant.... I rant... I'll get off my soapbox now.

[arran]

Our site is over 7 years old, has several hundred pages indexed by Google, hundreds of good inbound links from around the planet, but only one page is indexed by Yahoo.

Now this IS depressing. Definitely sounds like some sort of penalty.

Jaffstar, I see from another thread [webmasterworld.com], you used the re-review page [add.yahoo.com].
It's pretty disappointing they didn't even respond.
Did you previously have a decent amount of pages indexed or has it always been homepage only?

Let's assume for a minute both of you don't actually have a penalty - what are your linking characteristics?

- Number of links
- Rough PR of inbounds
- Ratio of homepage:deep
- Rato of one-way:reciprocal
- Anchor text variation
- Interlinking?

[morphisb]

I have this same problem with one of my sites, but it's not new. However, I created a new site about a year ago and it's still in development, I wanted to get a lot of content on it first before I had the SE's crawl it.

Well, this new site got fully indexed by Yahoo, and by Yahoo only. I still don't know how they discovered that site.

My other site, which has been around for about seven or so years, has gone through several revisions. For a while probably at least a year it went without updates, and I never kept track of how it did in SERPs until last March. Starting last March I went through the whole site, nothing big, and cleaned up the HTML, added some new content, and have gotten some quality inbound links (15-20).

This site, still will not get fully indexed, just the homepage. What's interesting is when I type the www.domain-name.com into Yahoo! the first two listings are my domain name but they have different descriptions, the first result is my META description only, the second is my META description and some extra on page text that Yahoo! pulled.

I sent an email to their search results help team (I think) but they never responded...

In the meantime, Google and MSN fully index the site, but of course, Google buries it in the SERPs, it makes wonder if somehow I got a penalty, I asked Google but got the generic reply...the site has gone through numerous changes over time, I've deleted pages, moved pages to different folders, redirect old pages to new location etc. Maybe I screwed myself over, but everything is much cleaner and organized, hopefully I can bring resserect this site but it's discouraging to see a lot of work get punished and not know why...

[Voxman]

Now back to 1,700 pages which what it was last November....yet affiliate and spam sites are doing very well... One I am montoring is a fully interlinking series of sites all owned by the same people and they have 225,000 pages of affiliate content in Yahoo. Good Job Yahoo (*dripping with sarcasm)

My attitude now is to go on as if Yahoo doesn't exist (that's what we had to do anyway for the last year). They wouldn't know how to crawl out of bed.

[BlackRaven]

"They wouldn't know how to crawl out of bed."

lol....so true.

[Voxman]

Now it's 15,900 wish I could get exicited but it's fluctuating too wildly... probably be down to a thousand tomorrow.

[jaffstar]

Did you previously have a decent amount of pages indexed or has it always been homepage only?

Yes, I had a few thousand pages indexed. The site also had about 20000 incoming 1 way links.

There is unfortunately no patern.

I have another site,which is dominating Yahoo, this site is new. New sites seem to do well.

[Smashing Young Man]

Odd. My site is just shy of a year old, and has on a couple of dozen, though high quality, inbound links and I have several thousand pages indexed in Yahoo and a few hundred referrals from them each day.

I get next to nothing from MSN, despite being well indexed.

I just got my first deep crawl from Google in months after being smacked with a duplicate content penalty after screwing up a 301 (I think that's what happened anyway).

I'm beginning to think SEO is a giant colossal crapshoot.

[abbeyvet]

I have a site about 5 months old with a little under 1000 pages of content that has been fully indexed by Yahoo, who send quite a lot of traffic. So it may be useful to know what sort of site Yahoo DID index and seem to like.

The site is in a fairly competitive area, one which tends to have a lot of similar sites all with much the same auto-generated information, but also a fair few with original content. This one has almost all original content, most of which already existed offline and was just made available online at that time.

It is added to and updated daily, at least a page a day is added, usually more, sometimes 10-12. These are being quickly picked up and tend to be in Yahoo and getting referrals within a few days.

It is not a commercial site, in that it sells nothing directly, though it does have adsense on all pages and a few relevant affiliate links here and there. While it is not a 'made for adsense' site, the fact that adsense exists was part of the motivation for getting all the content online.

It was submitted to the Yahoo directory for a free listing about 6 weeks ago, accepted and given sunglasses. But they had indexed it and it was doing well with them before that. It is also in DMOZ and maybe 5 other relevant directories. There are no more than a half dozen other incoming links (excluding ODP clones and regional Yahoos). Links were not sought (yet) and have been acquired naturally.

While Yahoo send a lot of traffic it is not the top SE referrer, twice as much comes from Ask Jeeves. In the last 2 months Google referrals have started to rise from a low base, they are running about about 90% of Yahoo this month, whereas last month they were 60% and in April less than 20%. All natural serps, no adwords. MSN referrals on the other hand are falling off as a percentage of the total.

The domain was registered about 4 months before the site was live, but there was nothing on the home page, but the content was being put online behind that.

I don't know how useful that is - I have other sites that do far less well in Yahoo, so need to sit down and take a long look to see can I do the same with them! I was not, to be truthful, thinking about Yahoo at all when making this site, more about making it Google-friendly.

[nzmatt]

It is also in DMOZ

Lucky you - how hit and miss is getting a DMOZ listing...

[arran]

Thanks SYM and abbeyvet - great to hear Yahoo being kind to a couple of newish sites!

As far as links are concerned, it looks like a small number of high quality inbounds could be the way to go. Are all your links on-topic or just from respected sites (or both)? I'm guessing the key is to get a couple of links from sites that yahoo loves (high number of pages indexed, spidered regularly and ranking well). Sounds straight-forward but we know Yahoo can be particularly fickle...

[sem4u]

Yahoo!'s crawling patterns can be a complete mystery at times! I have two new sites. One was crawled by Yahoo! almost straight away and now has at least some positions in the SERPs. Another site, where the keywords are much more competitive and can bring in a fair amount of revenue, has only two pages crawled so far. I would say that the second site has more links pointing to it.

Maybe the site just needs more links, maybe a Y! directory listing, etc.

[abbeyvet]

Are all your links on-topic or just from respected sites (or both)?

All on topic. A couple from very well respected sites which are not in the same niche but which link to articles that overlap into theirs. A couple from sites that are a tiny niche within the site's niche, again to articles, not to home page.

I am also starting to get people with sites in a smaller niche than mine to write articles and exchange links that way, I am offering to write articles specifically for some sites also. In fact that is one of the main ways I intend to build links. It'll be slow, but I think of higher quality.

One other thing. The site links out very liberally, but from within articles or beside them, not from a links page (there isn't one). These outward links would pretty much all be to respected and established sites or to sites in such a tiny niche that they are pretty much the only ones in it. Many of the sites linked to are non-commercial and are hobby or enthusiasts sites.

[jddux]

I have a few websites that I set up using subdomains, working great in MSN and Google, but Yahoo is not indexing pages within the subdomain, so for example it is indexing sub.mysite.com fine, but will not index sub.mysite.com/folder/ or any other files within subdomains, in google I have 13,000 pages indexed in subdomains and non in yahoo.. I compared with some other sites using subdomains and it seems like they are having similar issues, Have you ever come across this?

Pages within the main domain are being indexed fine.

[Tami_SEO]

Uh, one thing I know for sure is that if you have spaces (%20) in your full URL, they won't (or didn't) used to allow those pages. They also don't like dynamic urls that have? and other wildcards in the URL. This is only based on "submit it" rules rather than some other way into MSN. I know MSN has it's own submit page, but I'm not sure where submit it results vs other results show up.

Just my 02

[Vulcan315]

"Now it's 15,900 wish I could get exicited but it's fluctuating too wildly... probably be down to a thousand tomorrow."

I'm still at _1_ :-( . I've just made a few subtle changes and hope it helps.

Dave