Forum Moderators: open
* a human edited directory with good organization, little spam, but with a poor performance in terms of submissions, updates etc.
* or a fully automated directory with less organization, more spam, but with a high throughput for submissions and updates?
Please, don't rush to the answer. Give it some thought.
If you read the charter of this forum, a main point of directories is to act as a kind of seed for the "other search engines". The term "other search engines" seems to imply that the search engine is automated and finds all information by itself without help from human editors or users, e.g. Google.
The question is: how will directories stay useful in the future? Are they going to be "seeds" for the other search engines, or will they have a life of their own providing usefulness on their own terms?
Now with that said. The Web is so huge now that no general directory (ie. ODP, Y!) can hope to keep up with it. I suspect the days of huge general directories are numbered.
I think general directories need to position themselves to limited categories and limited sites (ie. top X amount sites we have found for Y subject that we consider "of quality").
I think the real future for directories is as smaller topic related directories. Perhaps the proper model we should strive for is that the large general directories and the spidering engines should channel traffic into seperate smaller directories covering a specific topic. Specialization as opposed to trying to do it all themselves.
I think general directories need to position themselves to limited categories and limited sites (ie. top X amount sites we have found for Y subject that we consider "of quality").
How do general directories review enough sites on each topic to decide what the "top X" sites are? That's the challenge.
I think the real future for directories is as smaller topic related directories.
I agree completely. The topic-related directories may not always be directories per se, though--in many cases, they're likely to be editorial sites that offer directory pages along with articles, forums, etc. (FWIW, that's I do on my site: provide annotated links in a larger editorial context.)
People are going to go to the properties that they feel offer the best targeted information. Human edited directories are the only way to go. Searchers are looking for one thing, answers to their queries. In a perfect world, a directory should present those answers in the first ten results without any paid advertisements or manipulation.
But, in the real world, that is not the way it works. Most directories are driven by the advertising dollars. That is what keeps them afloat. In my mind, Google is probably one of the leaders in how it serves results. Very little advertising and the unpaid listings get just as much exposure as the paid listings. Equal balance for all!
Here is a little snippet pertaining to Vortals or Industry Specific Portals...
Vertical Industry Portal is a portal web site that provides information and resources for a particular industry. Vortals are the Internet's way of catering to consumers' focused environment preferences.Vortals typically provide news, research and statistics, discussions, newsletters, online tools, and many other services that educate users about a specific industry.
As the web becomes a standard tool for business, vortals will join and maybe replace general portal sites like AOL and Yahoo! as common gateways to the Internet.
Given that the traditional directories of today are crumbling what is better:* a human edited directory with good organization, little spam, but with a poor performance in terms of submissions, updates etc.
Currently manage 14 sites
1 site - 23 listings in OPD
1 site - 15 listings in OPD
2 sites - 10 listings in OPD
the remainder all have 4 or better
In the past two months the submission rate of acceptance has improved dramatically.
My money is on OPD - first and foremost related to Google, AOL and Netscape and feeds many others.
Always tricky. Another problem is how do small directories also provide access to timely and name brand information. I think most topical directories would benefit from some sort of search backfill (secondary results) from a spidering engine. You never know what people will search for.
>they're likely to be editorial sites that offer directory pages along with articles, forums, etc.
Reminds me of Mining Co. before they became all commecially as About.com. Suite 101 seems to operate on that model. Editorially they might only have 20 links that the editor has hand selected on the subject, but those are selected with a lot of care for relevency and quality. It's an entirely valid model. Such sites make a great starting point for research and surfing. You are relying on the guide's personal recommendations and input, if you have a good guide it's priceless.
One mission that I have is to help small - quality amatuer sites get traffic. Many such sites are poorly optimized for spidering engines and/or will never be able to compete against large mercantile interests targeting the same keywords. As a topical directory I help such sites get found by surfers.
>or a fully automated directory with less organization, more spam, but with a high throughput for submissions and updates?
You still have to patrol it like a hawk. The spammers are legion, relentless and hard to thwart -- kinda like rats. :) If they get the upper hand the searchers will leave and you will end up with a glorified FFA site.
a fully automated directory with less organization, more spam, but with a high throughput for submissions and updates
If I had to choose, I'd go with the machines. But, actually, I think that all general web directories are facing functional obsolescence. They can't cope with the scale of the web as it is today, much less where it will be in 3-5 years. On top of that, JohnQ is learning a bit about search... still woefully ignorant, mind you, but at least he's now marginally aware of dynamic returns, ala google, fast, even firstgov. As he develops his search skills, he's likely to discard all but the most specialized, topic-related directories.
I tend to agree. Both Y! and ODP are being squashed under a huge backlog. There is no way of knowing what the backlog is at Y!, but at ODP the backlog of unreviewed is over 1,000,000.
Y! has elected to concentrate on the paid listings, while pretty much letting the rest of the directory fall further and further behind in relevance. Both have severe link rot - much worse in some categories than others.
In theory, a human edited directory is better. But that assumes that you have enough humans to maintain it. A poorly maintained directory is not as good as an up to date search engine. The all volunteer ODP sounds like a good model in theory, but they have been plagued by stories of self-serving editors for years - and true or not, it leads to a credibility problem.
In fact, if you do some searching on the net to get advice on how to get listed in ODP, it has become quite common to suggest becoming an editor in that category rather than the usual submission channels. Submitters to Y! do not have that option - so unless you have $300 you don't need, your site may never get listed.
I suspect that a combination of automated and human might work to some extent. One of the biggest problems at Y! and ODP is the sheer number of submissions. If some decent "pre-screening" software were available for the editors, that would probably cut down the backlog considerably. What you would need is a program that would spider each site, track it down to it's domain address, and compare it with others to check for duplication, spam, and mirrors. That would at least be a start. It could also be set to look for certain key phrases that are common in spam and affiliate sites.
<<What you would need is a program that would spider each site, track it down to it's domain address, and compare it with others to check for duplication, spam, and mirrors. That would at least be a start. It could also be set to look for certain key phrases that are common in spam and affiliate sites.>>
That would save a tremendous amount of time. It's too bad the ODP/DMOZ is not funded better.
I had to turn myself in yesterday because I noticed I was listed twice by accident. I really hated doing that, but I am always on my soapbox about being honest etc :)
However, human edited directories are great for analyzing clusters of web sites. That's great when trying to find related sites. A general directory will at least use the same technical standards for a wide range of topics, and to some degree maintain a uniform editing standars.
This would point to a possible future where the human edited directories are used as knowledge bases for search engines, and the improved search engines would be strong competitors to "fully automated" directories.
In plain words, my guess is that high quality directories have a better future than any automatic directory that accepts more spam and poorly classifies sites.
However too many DMOZ categories do not have editors - so there is no consistancy in specialisation per category, and too many sites do not get viewed (1 million is the rumour) within a reasonable time - so DMOZ cannot be up-to-date. DMOZ/ODP can also be very inconsistant in their multiple listing per site. Officially they say one listing per site, in practice many get more, and many should get more listings. The problem is, one editor can be very strict in this guidline and the another not, all for just or faulty reasons.
Most good, up-to-date and well maintained sector related directories have paid editors and are related to associations. Yet they miss the sites that cannot or will not pay the submission or association fee.
So you need an integrator of the Paid Association Directories, of DMOZ/ODP and of independant, quality sites/directories not available in the previous two.
Up until now, Google integrated this best in their normal search facility, by using pagerank.
Unfortunately Google puts a lot of weight on ODP/DMOZ though, indirectly through the high Pagerank DMOZ earned, but which is inherited down to too many categories, but also through suggesting ODP only categories [google.com] on top of normal SERPS.
I hope when Google comes with some categorisation/theming/sectoring above the normal search, similar to the glossary/labs [labs.google.com] feature things will be better. Certainly if they include a "directories" choice which does not only select from DMOZ/ODP.
I'm sure it depends on the cat but;
Submitted to DMOZ Tuesday, added today title and description unchanged, cat has no editor. [24hours behind Y! but we did pay them, still waiting for L$ UK].
Without doubt humans do it better!
But so long as ODP can operate on volunteers and (relatively) small overheads, there's no reason why it shouldn't survive, even its current form, for a good while yet.
And, realistically, it can (and will always) evolve with the changing market.
Don't get het up about size; that isn't everything, and even if a robot could be all-inclusive (as Alta Vista once was!), why would that be better?
However spammy your sites, however much you gripe that ODP won't list affiliate / mirrors / clones / deeplinks, etc ... just suppose it did?
However awful your clients, the 'all in' list would swamp it with dire deeplinks that earned nothing for anybody.
If the future for directories is quality, the future for sites is the same as it's ever been - ***CONTENT*** - there's no short cut.
Ask any 13 year old kid. They know. Spammers are mostly 25 - 34 - they still believe their own marketing!
Actually, no - it cannot, or perhaps will not. The thing about DMOZ is that it is not market driven, so it has no incentive to change it's ways. If you bother to read back over all the old messages going back to early 2000, you see exactly the same complaints then as you do now - unresponsive editors, self serving editors, dictatorial and unfair senior editors/staff, taking ages to get listed, out of date directories etc. etc. etc. Not one new complaint has surfaced in two years - just the same old ones that have never been resolved.
The only real difference is that now these complaints are coming up more and more often. That indicates to me that nothing substantial has changed inside ODP since AOL took over. It is essentially stagnant and not capable of making market adjustments.
The kicker; I used to manage a site for a friend, he left and set up on his own and I went with him and helped set up a new site. The old site got took over by someone else, it was not good and got dropped by ODP. I got an email from the editor [as I submitted the old site with my email address] saying that the site had been dropped, that he/she dropped it with some regret but felt it no longer met the standards of the ODP. Beat this [paraphrased] "I drive past everyday, I know you can do better than this, let me know if things change". How's that for service?
There are some excellent and helpful editors in many categories at DMOZ.
My concern is not with the editors, but that the editors in some categories are getting totally overwhelmed with submissions. I don't think this is peculiar to DMOZ, but with any free submission directory.
The problem with any volunteer run operation - directory or anything else - is that most people tend to gravitate towards the more interesting stuff. You will not find many people that get excited about editing thousands of webhosting sites when every one offers almost exactly the same thing - yet to be complete, a directory should have those sites listed.
We are looking at and evaluating upgrading our ecommerce storefront for one of our sites. I tried using the DMOZ directory - and the Yahoo directory - and both were so badly out of date as to be pretty much useless, so I ended up using Google and asking around in forums. And, oddly enough (or maybe not), the Google ads that showed up proved to he some of the most relevant to what we were looking for. The Yahoo Sponsored ads were almost useless, due mainly to the inadequate search that Yahoo does - far too many totally irrelevant sites were listed.
Imagine logging in everyday (or less) only to be greeted by 100 new submissions, 95 of which are junk, mirrors, and other assorted spam. Of those 5 that belong in the category, 2 of them bothered to read the guidelines and submit according to them. With some of the large numbers of unreviewed that have been tossed around (up to 1,000,000?), its got to be a situation something like that.
Any directory that is to stay really current, has probably got to be pretty small and have some financial incentive to maintain quality. Passion can go a long way, but it really does take a lot of work to maintain a directory.
It will forever be a catch-22. Payment for inclusion = omission of valuable information. No financial incentive (assuming editors are not independently wealthy) = less maintenance longer wait times, but better variety.
It will never be perfect, but will probably always be a combination of automation and human review.
If everyone would just let the editors and algos do their work and not try to manipulate them, listing times would be faster, Google search results might be even better, and it would save everyone a lot of work:)
Google now returns great results for any search term, indexes far more pages (therefore much more likely to find what you want), and is much quicker to use (no wading through multiple categories). I strongly suspect that the sites that appear high on google are the best sites for the search phrase - and that many of these sites are missed out or are incorrectly ranked on the directories.
Do we need directories?
Perhaps google can use its results/index to automatically create a directory using some algorithm.
It could automatically create categories based on the terms people are searching for - and list only the top 20 sites that would appear for that subject in its normal search facility.
Editors could then be used simply to remove useless sites and refine the process. I believe that Computers Do It Better!
That is a good point that I had not considered. SE's, especially Google, are much better now. I find that I almost never use any of the directory listings from either Yahoo or DMOZ. SE searches are usually much more relevant it seems - there is no "website quality" ranking in the directories, so you get a ton of crap for every good one.
I have to respectfully disagree when it comes to small directories for niche markets. I can go to a small directory about cooking, real estate, or travel to a specific area (among other things)and many times find a lot more relevant information. My wife uses small directories all the time for recipes etc. Many times this information could be on the 200th or 2000th page of Google just because the site is either not indexed or is not optimized. That does not mean it is not relevant even though the site itself may contain only a small amount of pages. I use these niche directories all the time and like the idea of "concentrated" information.
Of course you would probably expect something like this from me :)
It is imperative that directories clearly spell out what they want, and what they don't want. It is imperative that the consumer and SEO know these guidelines and follow them to the T!
The ODP is an excellent directory that has fallen under heavy fire from the voice of the SEO community. Its unfortunate that there is bias along with politics amongst some of the editors. This will happen in any directory of the ODP's size and especially becomes prevalent when it is run by volunteers and not paid editors. There is a big difference between the two.
Because of the sheer size of the major directory databases, results are going to suffer. General categories suffer the most. Its the focused categories that return the most relevant results. Now, getting to those focused categories can be tricky and most consumers may not know the process, so listings in those cats don't get the visibility they deserve.
I guess it all comes down to the search query. How experienced is the surfer in performing searches? Which logic do they use when entering search terms? That's the problem, its the user, not the directories. If I enter a 2 word generic phrase, of course I'm going to be presented with a bunch of results that may not be relative. If I now focus my search using 3, 4 or 5 word phrases, I will now probably see the relative results that I'm looking for. Sorry, I'm rambling...
Bottom line, smaller, quality directories are the wave of the future. Topic specific resources will do well when all is said and done with. Now its up to those topic specific directories to do everything they can to achieve visibility.
Watch the SERP's over the next 12 months. I'll bet you start to see more and more directory resources popping up in the top results!
I am wondering if this will be the case. With google looking at most "directories" as link farms (and rightly so), where does this leave the legitimate directories.
I am not going to suggest that google has ulterior motives in applying PR0 to directories (who could be legit competition for them), but I am concerned about how they would differentiate between the phoney and the real McCoy.
I believe the structure of the directory will have an influence on whether or not Googlebot sees it as a link farm. Most link farms have very specific footprints. I would think it is very easy for Googlebot to detect these and assign appropriate PR. If the topic were Blue Widgets and Googlebot found a directory targeting Blue Widgets but found links within the directory that led to other sites not relative to Blue Widgets then there would be an issue.
But, if Googlebot found the Blue Widgets Directory and a majority of the links within that directory led to Blue Widgets resources, then there should not be an issue. It all comes down to relevancy and the whole PageRank algo.
Let's not forget that Google is not the only search engine out there. Sure, it is the predominant force right now and a very important part of your online success, but there are many other SE's that have a share of the audience.
Right now, things are looking very good. I'll know more after the next major update.
To use an analogy: if this was the film world Google would be real good at finding Hollywood blockbuster films. In other words it finds you the most popular (boxoffice) results.
Those may not be the best films (results) because little known, indie, foreign and art house films would not rank so well.
Most of the top sites on the SE's got there through either a lot of hard work and study or because they hired a good SEO. That might not mean those are the "best" sites. They just had more money, time, talent, knowledge than the other. Certainly quality plays in there too, but only as part of it.
Now, to be fair, a high quality site will eventually rise up through the SERP's on Google. (Much to Google's credit.) But this could take a very long time -- many, many months, especially if the web designer knows little about designing for SE's.
A directory can have that same site listed and at the top of the serp's in a much shorter time. The beauty of smaller topical directories is that they can lead you to these lesser know gems much more quickly. (Sites that are on page 140 of the Google serps.)
I sort of treat each directory like it is a museum, each one has a unique "collection" of links which reflect both the quirks and mission of it's operators.
Pageoneresults, good luck to you on your new directory - I too will find out what google thinks on the next update ;)
SE contain far more results - so if your looking for a VERY specific topic chances are you'll find it on a google/fast/inktomi powered SE - but probably won't find it on Yahoo or Dmoz (depending on how specific your seach is!). Most SE's have crawlers that will find just about any site - including useful small personal websites.
Directories can't compete with this to the same degree - everyone producing small websites would have to know how to submit to dmoz, yahoo, lycos (.co.uk version) etc - so directories in effect leave out and penalise many of the useful-though-small websites!
The main advantage I see with directories are for new users to the web who don't understand that to do a 'search' you have to be as specific as possible - refining your keywords. The advantage of being able to click through to different categories to find what you want then becomes very useful.
The future for SE is assured, but for directories I think it will be a hard time. Most people use searches now (or the 'search' facility for the directory) - so the way forward seems to be continuing to integrate directory results into normal searches. The problem with this is that in the future normal automated SE results may eventually become more accurate and reliable than directories. These SE could then generate directories automatically using similar algorithms as they do for normal searches. Google uses Yahoo and ODP results to boast its own results. However it provides its results to other sites including www.BBC.co.uk website (without the Yahoo ODP results) - and results still seem good.
** I've not come across very many of the "smaller topical directories" - these sound interesting and useful. How do people come across them in the first place - related popular websites, magazines or webrings? Can you give me some examples please? :)
