Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo! Directory & SLURP

Correlation between a Yahoo! Directory listing and if the Slurp crawls site



8:38 pm on May 30, 2005 (gmt 0)

10+ Year Member


I've been reading WebmasterWorld for sometime now, but this is the first post I've made. I recently (late March) had a site listed in the Yahoo! Directory via Express means. It is a shopping site (won't get too specific) and at first I enjoyed great success with the terms that were provided in my description. However, I was wondering if people had problems having more than their index page listed?

I can not seem to get Yahoo to crawl the rest of my site, even though their cached index page is up to date. Google, MSN have no problems crawling it & there is no loose code from old css's. The site also has a full Site Map linking to every single page.

All that is listed on Yahoo is the Title & Description from the directory, and a current cached page. Is there someone or an email you can use to contact Yahoo (or is it even worth trying)? Also would it be easier just to remove my Directory listing altogether?

Sorry this is so long. Thanks in advance for any help or insight people might be able to give.



10:54 pm on May 30, 2005 (gmt 0)

10+ Year Member

You're not alone! I've got a few sites still waiting to fully spidered ... MSN and Google have the sites completely indexed already.

Any ideas?


10:55 pm on May 30, 2005 (gmt 0)

10+ Year Member

I never paid for inclusion in the directory. I just submitted my home page and about a week later, they were scanning my site (about 40-100 pages a day for a week or so). Now they will scan it almost daily. The number of pages varies between 50 to 3000 a day now. This weekend was really slow but earlier in the week I had a few days in a row where thousands of pages were scanned.

I would double-check your "robots.txt" file if you have one.


11:27 pm on May 30, 2005 (gmt 0)

10+ Year Member

Thanks for the replies.. I have no robots.txt!

That's the thing! I've submitted sites to their search before (non-directory) and they all got fully spidered. Is the directory a rip off or something? Mine's a commercial site, and as time's gone by, it's slipping farther and farther down the ranks, with no new pages being spidered. My income has dropped about 2000% from it (maybe more.. I dunno.. a lot anyhow).

Thanks for the help tho guys... anyone have any other ideas?



10:23 am on May 31, 2005 (gmt 0)

10+ Year Member

Do you define your canonical URL using a redirect on your .htaccess?

One of the things I have noticed on my sites that are not indexed is that Yahoo seems to have 2 instances of my index. If I search for just 'mydomain.com', Yahoo returns 2 homepages.

I know this has been mentioned as a problem elsewhere on the forum, have you looked into that?


12:54 pm on May 31, 2005 (gmt 0)

10+ Year Member

If your only getting your "index" page in the Yahoo SERPS you probably have a ban on your website (unless it is new). It seems no amount of begging to Yahoo will get them to un-ban nearly every site once they have decided to place a ban.

As far as the Yahoo directory, it used to be a popular place to get listed. As the SE's got better and Google rose to fame, the Yahoo directory became less an less used by surfers. It's value today is pretty much just in having the link to your site for other SE's PR ratings.

I have several websites in the Yahoo directory (one of which was banned by Yahoo in February - God and Yahoo only know why). I will NOT be renewing any Yahoo directory listings that require further payment as the value is much less than their annual fee (some sites are grandfathered before the annual fee was instituted). They should reduce it to about $29 one time with no annual charge to make the cost relative to its value today.


2:47 pm on May 31, 2005 (gmt 0)

10+ Year Member

Thanks for the messages andy & wally..

I only have one copy of my site showing when I do site:www.mydomain.com.

With regards to Yahoo! bans.. how does this occur? I *may* have overzealously submitted the site at one point (ie. once a day for like 5-10 days) because I was frustrated that the Directory wasn't picking up all the pages, but that's about it. There are some affiliate links on it, but each page has enough content on it to outweigh that I think. I also stripped off all alt tags on images, only one H1 per page, 1-2 H2s and maybe 1-2 H3s, I only target 1-2 keywords per page (aside from the index which targets about 5-6), and they only occur in the H1 and 2-3 times in the content text.

Completely baffled how you can pay $299, get listed and not slurped.. I could see it if they refused listing me, because then it would have something to do with the content, html etc.

Is there any value in removing the site altogether, licencing a new domain, putting all that content back onto the new domain and just submitting it to their SLURP once and only once? I don't get good results from google/msnsearch as is (although google has all my pages indexed, and msnsearch about 10%).

Thanks again!


3:02 pm on May 31, 2005 (gmt 0)

10+ Year Member

I dunno whether it would be worthwhile ditching your existing site because for a few reasons ...

1. You've paid for a directory link, you might as well keep it.

2. If it has been spidered and indexed by Google then it'll probably be sandboxed - but the clock is now ticking down to when it will be released.

3. Yahoo are typically slow to index, lots of people have the same experience.

Why not just start another site (without the same content) and see how it goes?


3:17 pm on May 31, 2005 (gmt 0)

10+ Year Member

Thanks for the help.. I might just start up another site, and have that link to the content of this one.. ie. This one is a price comparison/shopping site, so with a new one, maybe target the 4 or 5 main categories and tie it all together that way. That way I won't lose the Google sandboxed pages (any idea how long it takes to get out, or is that a dumb question because no one knows?), and I should get a quick high ranking on Yahoo for the main categories of the new site.

My real concern is for the XMas season, just want to make sure pages are showing up then (both in yahoo & google)

Thanks for the help though, you've given me a lot to think about and some ideas here.

Best regards,


3:57 pm on May 31, 2005 (gmt 0)

10+ Year Member

Some argue that the sandbox is fictional, but for me it has been very real ... even for fairly non-competitive phrases. I have had one site 'released' or cleared from whatever filter used to sandbox a domain, but it took a year to happen. Personally I would concentrate on Yahoo / MSN for the mean time, keeping an eye on Google for the future.

One thing I would say about your directory listing is that your homepage will appear in the Yahoo results with the same title used in the directory. If they have given you a good title for your directory listing, then that will be to your advantage. However, more often than not you'll be listed with your company name or with a title that won't really benefit your ranking. So although a directory listing in Yahoo is a solid enough investment (?), in some ways you are actually not helping yourself.

The only domain I have listed in the Yahoo directory ranks well in their search engine, but that site has been around for years and has thousands of backlinks. Even though it is listed in the Yahoo SERPs with it's company name for the title ... it still does well.

You may find that starting another site without a Yahoo directory listing is the best thing to do, especially if you are going after some competitive phrases where you will need control over your title tag.


10:16 am on Jun 26, 2005 (gmt 0)

I agree entirely. I had a similar experience recently and this problem basically emphasis just how important it is to check the robots.txt file.

Featured Threads

Hot Threads This Week

Hot Threads This Month