How to Get Yahoo Slurp to Crawl and Index a Website

Top Considerations for Getting Indexed Well in Yahoo

[windriderme]

I am trying to figure out how to get my site crawled by yahoo. Right now both MSN and Google have the current pages for my site, but yahoo's listings are months old. They have not even registerd a title change that was made back in March. From looking at the stats offered by my site Yahoo's bots want nothing to do with it. If I was not listed in the directory I would believe they had ban my site. Does anyone know how to get yahoo to crawl a site besides submitting the site again (which I don't want to do incase they decide to ban it for to many submisssions). Thanks for your help in advance.

[martinibuster]

• Make sure you have links from websites that are frequently updated, and that slurp is known to like. I speak from experience. One of my content-rich sites has a single link from a good site and is hogging spots in the top five in Yahoo. It took a few months to get there, but it's jammed in there now and not moving.

• Make sure you have good original content that is unique to your website (affiliate feeds need not apply)

• If your site is ecommerce, make sure that you have enough content on the page for indexing.

• Make sure your title and meta tags are all different

I have another content-rich site with a ton of .edu's and mom & pop sites in it's backlinks, and it's not indexed well at all by Yahoo. I added a sitemap to it last week and Google indexed over seven thousand additional pages of 100% original content. Yahoo doesn't yet know it's there.

I'm fairly certain it's because the kind of websites that comprise my backlinks don't change very much, are sleepy or don't have a lot of good backlinks themselves, and Slurp doesn't favor them.

So I'm going to experiment with the backlinks and see what happens.

[arran]

martinibuster,

Make sure your title and meta tags are all different

Do you mean on the same page or between pages?

Thanks,
arran

[martinibuster]

From page to page. Some people with problems getting indexed have the same thing throughout the entire site.

[Shurik]

Martinibuster, maybe sites don't get indexed because Y wants us to pay for it. My new site does not get listed beyond 1st page although a have some decent links in and out. Slurp crawls the site but only home page shows up with the site: command. I don’t mind paying $300 for "submit express" to get 10 pages listed but charging $0.15 per click on top of it is just a plain rip-off! Does anyone has any evidence supporting the theory that getting into Yahoo directory helps in getting fully listed in their index?

[martinibuster]

I've never paid for a Y directory listing, and don't do sitematch, not even for clients. No correlation in my experience between directory listing and ranking well or badly.

In my experience, it's the the links plus the content. But I have sites that do well with minimal content, pretty much because of the links. Like I have said before, one really good link is enough to push a good content rich site up there. One link.

[Shurik]

A year ago my other site had no problem getting more then its index page listed in Yahoo index. Another site I set up a month ago on a sub-domain of an old domain took off in 2 weeks even with very few links. However my new site on a newly registered domain (3 month ago) is stuck at just one page. IMO it is reasonable to assume that they’re waiting for some “signals of quality” in GG’s terms. Perhaps quality and quantity of the incoming links is one contributing factor. But I wonder if getting listed in their directory is sufficient to indicate acceptable quality without aquiring ton of links. I hope someone could comments on it before I go waste $300 on their directory.

[sufyaaan]

Creating an RSS Feed for a site and adding it to My Yahoo! is one of the fastest ways to get a site indexed within 24 hours.

[edited by: martinibuster at 4:42 pm (utc) on July 12, 2005]
[edit reason] No URLs Please. See Tos. Thanks. [/edit]

[Frequent]

Yahoo is even more mysterious than $G for me. I have one site with minimal backlinks as well as some of the dreaded "affiliate feed" content that Slurp has indexed the heck out of. I get around a hundred referrals from y! a day and $G still has it sandboxed so they give me only 1 or 2. A couple of my other sites are fully indexed in $G but index page only in Y! And a couple other sites that are fairly well indexed in both. None of my sites are well indexed in MSN, one or two seemingly random pages max.

Freq---

[martinibuster]

Frequent,
Do all of your sites get spidered by Yahoo (that's the topic of the thread), and why do you think they do?

[caveman]

Similar to what MB says, my opinion is that crawl patterns may have something to do with it.

Also, I think that Tim has indicated that sites with good/better indicators of quality (e.g., backlinks) tend to get crawled more deeply.

But I'm not sure that either of these explain completely why some sites are not being crawled or featured much at all ... or why some sites have some subpages shown in the SERP's while other subpages of equal value are not shown.

My guess WRT sites having little or no luck in Y! is that it has something to do with site or IBL elements that Y does not like. An example would be like MB's comment re META tags. We had a site that could not get ranked in Y! despite doing well elsewhere. We did a site review and decided there was an issue with too-common META tag elements across pages. Once fixed, the site flew into top spots in the SERP's.

[Shurik]

Slurp does "lazy crawl" of my new site - mostly index page on daily basis. Occasionally is does a deep crawl. The site is doing well in MSN and is sandboxed in google. All 89 pages hand written with unique content and pics. Have bunch of links already, all non-reciprocal including deep links. Meta tags are distinct between pages however most are the same as page titles. Maybe they don’t like it. Will change and see what happens.

[Frequent]

martinibuster-

As I mentioned not all of my sites are well indexed by Y!. The ones that are well indexed by both G$ and Y! I attribute to good backlinks from a variety of sites. The anomoly is the first site I mentioned which is still very new (less than 4 months) with minimal backlinks but is extremely well indexed by Y!. I have not been able to figure out exactly why Y! has crawled and indexed this site so deeply. The site in question was heavily crawled by slurp literally within 2 weeks of going on-line. At that time I had less than a handfull of backlinks. The site consists of primarily unique content that is updated a few times a week. Slurp visits for a deep crawl about twice a month at most.

My first best guess is that one of my backlinks just happened to go live when Slurp was actively deep crawling that site. My second best guess is that I managed to dodge any sort of Y! "sandbox" penalty (by receiving no more than 2 or 3 back-links per week.)

I think a new site with a ton of backlinks right off the bat must trip a lot of flags with Y!.

Freq---

[martinibuster]

Hey Freq, that's really interesting. I wonder if just one of those links to your new site was enough to trigger the deep spidering.

I experimented with unique IP addresses and that hasn't made a difference. I still lean to the backlinks as a reason, but others have privately suggested that a site living on a server with hundreds of other sites may slow it down, but I'm not sure about that.

For what it's worth, I mentioned above that I had a site that didn't get adequately spidered. Well, I gave it a link from a site that I know Yahoo likes, and about 500 out of 7,000 new pages were recently indexed. Google indexed more. I'm of the opinion that the quality of backlink (and I use the word "quality" loosely) is important. I'm going to add more links and see what happens.

[Frequent]

I look forward to taking advantage of my freakishly Y! friendly new site to get some of my other sites properly indexed by them as well. I only hope that G$ gets around to that site soon as well. It promises to be a good traffic generator once G$ gets ahold of it.

I also had an affiliate site put up shortly after the aforementioned site that was shaping up to be very popular with Y!. Unfortunately due to some "issues" the entire site had to be dumped and started a-fresh. This site featured an amazon feed script (had to try it, it was free;) and slurp had gone through about 10K pages over the course of two days. How slurp found that one is a complete shocker. I literally only had 2 or three forum sig backlinks on a not-to-popular forum.

Needless to say I will make a fresh post or two there when I get my next project ready to go. If it has the same effect I have a new best friend.

Freq---

[martinibuster]

If it has the same effect I have a new best friend.

Yeah, isn't it strange the way we make friends?