Are Directories Dead in 2016?

[Andreas8]

(Sorry if this has been answered earlier.) I am thinking of buying a domain, (which is not cheap) and has the keywords of; myniche + directory.com.au in it. On the domain I plan to build out a directory, probably using a word-press theme and word-press directory plugin. It will be an industry specific directory (only for that profession) and will be hand edited by me. It will be exclusive, that is, all listings will be manually approved and no spammy links or bots allowed with the hope that it add value to the user experience. My plan for the directory site is long term as I am qualified in the profession and I like SEO. It will begin with free basic listings, also hand edited by me and I will at some stage allow paid listings for reasonable fee.

Directories have a bad name currently in the SEO world, since Panda and Penguin and Poodle (only joking about last one) smashed many of them out of the park for good reason. That bad reputation may stop some in my profession from purchasing a listing or even wanting a free listing. But directories seem to have an influence on local search which could be the value that I add to the business that wants to list on the directory. e.g www.yellowpages.com.au has a PR of 7.

My question is: Are directories dead? Are directories good or bad for SEO in 2016 and moving forward if they are not spammy? Do directories add value to a business? Therefore should I go ahead with my plan?

Thanks in Advance :-).

[edited by: aakk9999 at 10:13 am (utc) on Jul 6, 2016]

[jmccormac]

I'm actually surprised directories still exist. I had thought their day had long gone with the turn of the century and the advent of Mr. Google.

Google only started murdering directories in the late 2000s. Basically they were getting traffic and advertising revenue and after Google scraped them for their content, one of the asinine Animal Farm kludges hit many of these directories. Basically a new directory subpage would have very little content and some directory owners may have auto-generated the schema using off-the-shelf directory software.

The DMOZ backfill schema is probably responsible for some of these SEO problems as sites relying upon DMOZ would have a largely similar internal link graph.

Regards...jmcc

[treeline]

I was originally inspired to start a niche directory by over a thousand emails I'd saved for the link to a cool resource. It reached the point where I couldn't find the right email anymore....

There's lots of cool information super relevant to a niche that doesn't register as related to a search engine.

[tangor]

I admit to keeping ONE directory alive ... and updated, etc. But it is also LIMITED to the top 100 in that niche (my pick). Curiously, there's enough folks out there that trust my instincts to keep it bookmarked and linked. :)

But I have no desire to do another these days.

[martinibuster]

Google only started murdering directories in the late 2000s.

Google didn't "murder" directories. Google's emphasis has always been on answering a search query. Sending their users from a page of links to another page of links is a poor user experience. That's all it is. Directory owners who didn't realize this and change were left behind with their "search engine friendly" nothings.

Web publishers like WebEnthusiast who understood the importance of user experience could think it through and evolve a variant that has added value.

All the travel/review pages with hotels, they are nothing else then a directories with products:

You nailed it WebEnthusiast! :) Yelp is a directory and they haven't been "murdered."

I've said this before and I'll say it again, focus on the user experience and your marketing efforts will align with what Google's rewarding in the SERPs. That is a form of SEO that I call User Experience Marketing.

[tangor]

SEO is a catchall term that really does not define what NEEDS be done: USO (User Site Optimization) ... but that would conflict with another acronym and might cause REAL CONFUSION. (sigh)

[JAB Creations]

I agree with topr8 that doing reviews for sites is a good idea though the trashier more crowded and especially monitized-looking a website is the less I trust it.

If you want to do a review-directory website and earn some money you can do paid-reviews though you need to explicitly label such reviews as such (like advertisement logos, small though still identifiable) and not allow the owner/operators/etc of the site begin reviewed to do anything except read the review after it is published. If they disagree they have the option to take constructive criticisms, improve their site and then ask for a refresh-review no sooner than two or three months. When you hold to a stricter platform though are still flexible it shows objectivity and that will lend the idea much more credit to visitors.

John

[ergophobe]

All the travel/review pages with hotels, they are nothing else than directories with products: Hotels. Hotels do not need their own pages/sites as the directories bring them enough clients.

This exactly the industry I was referring to. Trip Advisor, VRBO, AirBnB are all directory sites. Oh sure, over time they've added features - direct booking, management tools for lodging managers and so on. But in essence, they are directories.

Our local tourism board also runs a directory. It's more your classic directory - just a listing. No reviews. No booking ability. No online calendar. But they have a strong presence at trade shows, run lots of print ads, have a strong online advertising presence and deliver customers

This is the main thing
Old School: Directories deliver links
New School: Directories deliver customers

If you're delivering links, you're dead. Remember, that only worked because Google was too dumb to be able to figure out what you were doing. Google didn't murder you so much as unplug life support on an idea that was as dead as the Yahoo! directory was the day Google patented page rank.

If you can deliver customers and can prove it with tracking (which means letting the people in your directory attach UTM parameters), you have a cash cow and will *often* outrank the businesses you list. There is a reason that Trip Advisor will often outrank the property itself for many searches, especially if you exempt pure brand searches - it actually provides more value through reviews, ratings, context (what else is nearby) and the simple fact of having a third-party opinion. Users like that and click and read and stay. Google notices that and ranks it.

I'm not, by the way, denying that a given directory once provided value and now doesn't rank. Before Alta Vista, the only way to find a relevant set of sites on a topic was to find a directory that curated them. There was real value there. But that model just doesn't inherently have value unless you have something to add that a robot can't do.

[smilie]

Guys, read and reread what @jmccormac and @WebEnthusiast said.

I still have one niche directory that' practically dead. I am trying to revive it right now.

To the OP. Directories are not all dead. But dummies at Google are actively trying to kill them, because all they see is around is spam. Any good directories are being killed as collateral damage. And no, Google doesn't provide "better results" than directories, what Google provides is a list of brands in our niche on page 1 which is USELESS and really only maybe 3 out of 10 of those big brands deserve to be there. Google's algo is brand-heavy , and believe me if I say Walmart, Home Depot and Amazon are NOT authorities in our niche (we have at least 5 other multi-billion dollar companies, EPA and several .gov by the way).

Be prepared to spend thousands of dollars or spend your time (which is also worth money) to get something up and running. And then be prepared for a floodgate of spam , poor quality and unrelated sites and robot auto-submissions. (maybe if you don't do business in certain countries consider installing IP blocker and block several countries and well known proxies entirely, will save you grief of cleaning up spam).

And then what you are going to find is that list of sites with a paragraph of content is not enough and you are not ranking anywhere.
So think about your content strategy - and that gets yet more expensive.

[EditorialGuy]

And no, Google doesn't provide "better results" than directories,...

Spidered search engines don't provide better or worse results than directories. They merely provide a different kind of results (more granular, less "broad stroke").

[martinibuster]

But dummies at Google are actively trying to kill them, because all they see is around is spam

Read what I posted above. It's not about spam. Consider that not even DMOZ ranks anymore. The simple fact is that it's about user experience. Period.

Sending their users from a page of links to another page of links is a poor user experience. That's all it is.

[Storiale]

@ergophobe is correct. Does it produce results and the definition of results = $$$$.

[glakes]

This thread belongs in the appropriate directories forum located at [webmasterworld.com...]

[IanCP]

Being busy, something I had originally forgotten about this topic. Now with time to spare, I just Googled a recipe for tonight where I had forgotten some previous details.

Yep, in the listings there were several directories.

I can't speak for others, but I would much prefer Mr. Google sent me straight to the original site rather than through a wholly pointless intermediary step. Over the space of five minutes that happened to me three times.

I find it very, very annoying and wholly unjustified. As has already been said:

USER EXPERIENCE

[smilie]

>> @martinibuster: It's not about spam. Consider that not even DMOZ ranks anymore. The simple fact is that it's about user experience. Period.


Sure. User Experience. The Web is not about pretty things. Yet, all major search engines employ thousands of cheap rankers to rank the web. And those rankers have seconds to rank your site (including directory) , and they also have guidelines. This is their "social blah user experience" algorithm. So. If your site doesn't appeal to rankers, you tank. AND brands have a go and get an A for a lot of searches.

So, it is about "lowest denominator user" experience. Not content, not QUALITY of CONTENT anymore. 5 seconds of fame, if you look good you get ranked.

That's why directories tank. They don't look flashy. Some of them are even 15+ years old, in industries. But yet some of them link to sites that actually make sense, not Walmart, Amazon and Home Depot.

And the second , obvious, reason is because spammers scrape DMOZ like there's no tomorrow to auto-generate their stuff. And their stuff is sometimes so good Google can't distinguish between spam and not spam. So G, I am sure, built something into their actual algos to identify directory-like pages and demote them. And DMOZ itself is a collateral damage of this.

Let's examine this. What is Google? A collection of on-topic links. Which is also now human-checked. A giant, keyword-based directory.

[smilie]

User Experience, er?

In some niches you actually have to do this:

"search keyword -walmart -homedepot -amazon -brandA -brandB -brandC"

to actually get decent results. 2013-2015 were the worst years in this, it is getting better now but not by much and very slowly.
If it was generally only about "User Experience" this would not be an issue.

[martinibuster]

Yet, all major search engines employ thousands of cheap rankers to rank the web. And those rankers have seconds to rank your site (including directory) , and they also have guidelines

Human evaluators have absolutely nothing to do with not ranking directories.

I run online communities and it is forbidden for one member to tell another member to "google it' because it's a poor user experience. People come to my forums to obtain the benefit of someone else's experience. Telling someone to "Google it" is not an answer. Similarly, an algorithm that tells a user to choose from a page of directory links is also a crappy user experience.

It's a common sense fact that giving a search engine user a page of directory links as an answer is a poor user experience. A good algorithm will be able to choose the actual site you need to see. Telling a search engine user to pick a link from ten links, then presenting them a directory page with ten more links is well, stupid.

Human evaluators make mistakes
You are 100% correct with your implication that human evaluators can make mistakes or are biased. And the researchers at the search engines already know this. Thus researchers have used human evaluators to study the incidents when multiple evaluators rank websites differently. This research was done years ago and was compensated for. I encourage you to read about these studies because your understanding of human evaluators is heading in the right direction but there's a lot you do not know.

The search engines and researchers understand that human evaluators are subjective and this has been a topic of research to tune their algorithms to actually surpass the ability of human evaluators. The researchers used the conflicting human judgments to teach the machine and the machines learned how to make better evaluations than the human evaluators.

You are mistaken in thinking that human evaluators are calling the shots. The human evaluators are simply creating data that is fed to the machine to learn from. This point is very important to understand and is why I encourage you to understand what is REALLY going on because otherwise you are basing your business decisions on ideas that are invalid.

[EditorialGuy]

It's worth noting that directories are created by "human evaluators." People who don't trust other people's subjective judgments shouldn't be using (or touting) directories.

[NickMNS]

You are mistaken in thinking that human evaluators are calling the shots. The human evaluators are simply creating data that is fed to the machine to learn from.

The data from the human evaluators is not even fed to the ML algos to learn from, it used for validating the output, to ensure that the output is the one desired.

[martinibuster]

If human ratings aren't being fed to machines to learn how to rate, then I would be most interested in hearing what you have read that is being fed. What data is used to teach a machine to rate websites?

Machine Learning begins and ends with data. There's nothing to learn from without data to analyze. If you read the research papers that mention human raters you will see that the data is used much as the click log data is used, as something to learn from. Human raters are also used for image search.

In the famous article in Wired about the Panda algorithm Matt Cutts and Amit Singhal make a direct reference to this machine learning process where data generated by human evaluators is fed to the algorithm which then learns from it in order to scale the process of rating websites independently and better than humans. I edited the interviewto focus on the machine learning process. Read the entire article here. [wired.com]

Wired.com: How do you recognize a shallow-content site? Do you have to wind up defining low quality content?

Singhal: ... we used our standard evaluation system that we’ve developed, where we basically sent out documents to outside testers. Then we asked the raters questions like: “Would you be comfortable giving this site your credit card? Would you be comfortable giving medicine prescribed by this site to your kids?”

Cutts: There was an engineer who came up with a rigorous set of questions, everything from. “Do you consider this site to be authoritative? Would it be okay if this was in a magazine? Does this site have excessive ads?” Questions along those lines.

Singhal: And based on that, we basically formed some definition of what could be considered low quality.

Machine learning only works when it has a data to learn from. That's one of the purposes of the human raters. Not the only purpose, but one of them. From the human ratings Google was able to come up with classifiers that the machine could use to make quality judgments.

Wired.com: But how do you implement that algorithmically?

Cutts: I think you look for signals that recreate that same intuition, that same experience that you have as an engineer and that users have. Whenever we look at the most blocked sites, it did match our intuition and experience, but the key is, you also have your experience of the sorts of sites that are going to be adding value for users versus not adding value for users. And we actually came up with a classifier...

As I mentioned above, human raters judgment conflicted and this was an important data point for search engines to study, to come up with a classifier that was more accurate, more consistent than a human.

I am not at home at the moment to give WebmasterWorld links to research papers but they're easy to find. You're all search professionals (many of you at least) and know how to search for PDF files, right? I'm no longer handing this information over on a platter for everyone's convenience so I'll just point all of you in the right direction and you can research it yourself.

[NickMNS]

I 100% agree that ML starts and ends with data. Where I have trouble is to suggest that the data has all been reviewed by humans.

With all do respect the article you use to support you argument is an interview with two Google spokespeople hand waving their way through an intentionally vague explanation of how Panda 1 worked.

The volume of data required to produce a reliable algo at the scale of Google is enormous. There is no way that every positive website and negative website in the dataset was reviewed by humans using the Google Site review criteria. More than likely, a categorization was done using other methods, the top positive sites were selected as positive, the worst negative sites were selected as negative and anything in between that was ambiguous was tossed out of the dataset. Then the dataset was divided into two (most likely 3) parts, the training set, the test set (3rd would be a cross vaildation set). The algo would be trained, then run against the test set. Then a subset of the results of the test would be sent to the reviewers for manual review as a last check.

My point with all this is that when comes down to Google and other major search engines, human intervention accounts for very little.

[martinibuster]

the top positive sites were selected as positive, the worst negative sites were selected as negative

Think this through using your own statement quoted above. How were the top positive sites selected?

the article you use to support you argument is an interview with two Google spokespeople

Those are not Google spokespeople. Gary Illyes is a spokesperson. He has a journalism degree and is in fact nothing more than a spokesperson.

Matt Cutts and Amit Singhal are two of Google's top engineers. Amit Singhal was the head of the department directly in charge of the ranking part of the algorithm. It is a mistake to regard either of them as mere spokespersons, especially with regard to Amit Singhal. One couldn't find a more authoritative source of information about the algorithm than him.

Again, I encourage you to use your search skills(filetype:pdf) to find the papers and read up. I refuse to hand it over to the general public on a platter. I will only point in the right direction and leave it up to you to do the trivial research. And believe me, the effort to find this information is trivial.

Good luck.

[NickMNS]

I know who Matt Cutts and Amit Singhal are, my point was that in the interview they are acting as spokespeople, releasing to the public the company's scripted message.

I have experience with machine learning, much of the content on my website was created using various machine learning techniques.

Yes, at one point you need to start with a human classified data, but by the same logic you would need to invent a computer before you could start classifying the data.

When using Panda as an example, Google already had a pretty good classification of high quality and low quality websites. It is not like Google was unable to determine great from pure spam before Panda, they did not need human raters to go through every site or webpage in their dataset. If they did we would still be waiting for it to role out. ... wait? ..maybe thats is why we are still waiting for a Penguin update!

Matt and Amit, never say in the interview that human raters evaluated each and every site in the dataset. They simply state that human evaluators are used in the process.

However, John Muller in a recent hangout, possibly the last one stated explicitly that they don't use human raters to evaluate websites, they uses them only to evaluate the effectiveness of changes to the algorithms.

[martinibuster]

However, John Muller in a recent hangout...

John Mueller is just a spokesperson. I'm surprised you discount Amit Singhal yet cling to something Mueller said.

Matt and Amit, never say in the interview that human raters evaluated each and every site in the dataset.

Quality raters were used to create baseline training data. It's where the classifiers come from. Once you have the classifiers the machine can then use that to rate. Human quality raters are used in the process to compare between machine and human ratings. As I mentioned above, instances where human raters conflict are an important point for research, finding out where humans made the mistake and training the machine to not make those mistakes. This is part of the research.

So now you agree that human evaluators evaluated some of the initial data used but are quibbling that they didn't evaluate every data set? How about you just concede that you agree with me that the initial data set was based on human raters and that the quality raters were used for other purposes too? Because if you do then we'll be in complete agreement because that's what I am positing.

However, John Muller in a recent hangout, possibly the last one stated explicitly that they don't use human raters to evaluate websites...

Well yeah of course. That's a typical Mueller statement that confirms the obvious while rarely stating anything useful. Of course human quality raters don't evaluate every website ranked in the SERPs. But we all know human quality raters look at websites. What else would they be evaluating?

When using Panda as an example, Google already had a pretty good classification of high quality and low quality websites. It is not like Google was unable to determine great from pure spam before Panda, they did not need human raters to go through every site or webpage in their dataset.

No, they did not. Previous methods were based on statistical analysis. These are things like averaging how often search queries appear on a page, number of words on a page, noticing patterns typical to spam pages. Panda goes beyond statistical analysis.

Re-read the Wired article and you will understand that Panda is about the actual quality of the page and not about finding statistical outliers. Two different things. They took human quality rater data and created classifiers from that. Those classifiers were a way to imitate the "intuition" that search engineers and human quality raters used. Read the article again.

Then read this PDF [arxiv.org], specifically section 6.2 to understand how human quality raters are used to create a data set. There is so much more on how that works.

[piatkow]

It's a common sense fact that giving a search engine user a page of directory links as an answer is a poor user experience. A good algorithm will be able to choose the actual site you need to see. Telling a search engine user to pick a link from ten links, then presenting them a directory page with ten more links is well, stupid.

A directory that gives me a comparison of different B&Bs at my resort and lets me sort by price or distance from a place, or even eliminate those that don't have single rooms, gives a much better experience than ploughing through dozens of individual sites.

Directories work when they add value.

[jmccormac]

As I may have mentioned, 've been working on a full new gTLD web usage and development survey (approximately 17 million domains). It might be interesting to see how many web directories are appearing on the new gTLDs. But there is another very interesting aspect of some of thew new gTLDs and one of them, .KRED gTLD seems, in some ways, to be the next step in the eveolution of directories. Arguably, the same could be said for .REALTOR which also has good usage. The .KRED business model seems a curious one but managed TLDs could, if implemented well, provide a new life for the idea of directories.

Regards...jmcc

[NickMNS]

@martinibuster
The paper you linked to refers to a study conduct in an academic settings, its goal was to prove the value of a specific algo used for classification. They state that in total 14000 triplets were used and evaluated. This is well within the ability of human raters to evaluate. The problem is in the real, world working at scale, when you need many order of magnitude more data to produce acceptable results.

For image and search and ocr Google came up with the brilliant method of using the reCaptcha results, thus using an automated system of human rating at scale. But this is not used for web search. Is far as I know there are no captchas showing webpages saying "choose all that are spam".

I am not saying human evaluators are never used for feeding ML algo. What I am saying is that it is costly, time consuming and prohibitive when working at scale, so other methods need to be found to pre-classify the datasets, when working at scale.

[Drew]

I also vote that most directories are dead. Better to build Rand's 10x content however having visibility in sites where people are actually visiting would still be beneficial. (IMHO)

[jmccormac]

Who is Rand? :) Directories provide a good structure for some kind of sites however the focus on content for the sake of content is wrong.

Regards...jmcc

[FranticFish]

@jmccormac

Can I ask you to elaborate on why you think that .KRED is interesting for the future of directories? The extension was a totally new one on me. From what I can see it's a proposed proprietary extension for a brand that measures social influence. ICANNWiki indicates that the company might have some sort of application process for domains so acceptance might indicate a level of trust that could be relied on my others (i.e. Google) but that's all I can find out. Do you have any further information or thoughts?

[jmccormac]

It isn't so much a TLD as a directory using a TLD as its main selling point. I'm not sure about the process of getting a domain name but it appears that it is free for the first year with the registration fee being paid from the second on. The pages seem to be what would be considered one-page landers if they had appeared in other TLDs. But they provide links to the registrant's main website and, in some cases, a means of contact. It is not quite Facebook but it is similar in the emphasis on people/influencers. Directory builders generally try to come up with an ontology or structure to categorise sites but there is possibly a categorisation and organising process at work with this TLD that is similar to how directories work.

Regards...jmcc