Forum Moderators: open

Message Too Old, No Replies

Google and Dmoz

         

lloyd

3:20 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



Can someone please advise me what the current situation/relationship is between Google and Dmoz.

Other than Dmoz providing the results for the Google directory.

Is Google still using the Dmoz listings in the organic results. i.e. the Dmoz description in the Google algorithm.

Please advise

Spica

4:59 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



Related to lloyd's question: What happened to the concept that Google used to start its deep crawl at dmoz? Is that still true? Or is the spidering now constant, with the bots going from site to site endlessly, never starting over again at dmoz?

choster

6:13 pm on Apr 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google has licensed ODP data to use in the Google Directory. As far as I know, there has never been anything more to the relationship than that (i.e. they do not have any other business relationship. ODP categories link to the equivalent Google Directory categories if they exist as it does to Thumbshots, and Google links back to ODP per the ODP license).

Also, I don't recall that it was ever true that Googlebot always started with dmoz.org. ODP is the largest web directory extent, which means it would be a logical place to do so, but which also means that Google will find its way there sooner rather than later, and that as a trusted and frequently updated source it would be crawled regularly and deeply according to their algorithm.

I have not seen ODP descriptions in Google search results in some months, nor links to any Google Directory category where a url might be listed.

Genie

6:48 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



The change took place on 29 March 2004, when Google rolled out its new look (at least for all its English-language sites.) The directory tab was removed from the front page. At the same time links to the directory and site descriptions from the directory disappeared from the SERPs.

ILuvSrchEngines

8:31 pm on Apr 21, 2004 (gmt 0)



>At the same time links to the directory and site descriptions from the directory disappeared from the SERPs.

But the heavy weight that Google gives to DMOZ sites remain. They just hide the DMOZ link references so it is not so obvious. I think the heat they were taking on webmasterworld is the reason they hid the DMOZ references.

choster

9:33 pm on Apr 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Google has stated repeatedly that links from ODP are not given any heavier weight than any other site. Moreover, while the top-level dmoz.org and "become an editor" pages have been PR9 or PR10, the vast majority of sites are listed several branches deep into the directory and have commensureately lower PR.

Genie

9:43 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



> think the heat they were taking on webmasterworld is the reason they hid the DMOZ references.

I don't wish to hurt any feelings here, but I suspect that the views of WW regulars matter less to Google than the responses of its millions of users.

Search engines overtook directories in popularity several years back. I can see the thinking in the Googleplex. The links featured on the top page of Google should be to the most used Google services. The directory tab was replaced by the link to Froogle.

If the links in the SERPs to the directory weren't being clicked on all that often, I can see why Google would want to save space in the SERPs by removing them too. After all they have the 'Similar pages' link, which does much the same job.

Having removed the links, they would have to stop using the site descriptions in the SERPs as well. Otherwise they could be in breach of the ODP license.

[edited by: Genie at 9:48 pm (utc) on April 21, 2004]

Symbios

9:46 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



I think Genie summed up the situation nicely, I also recollect GG saying that that the directory tab wasn't used that much (or words to that effect) so it made sense to add something that may be of more use to people using G.

ILuvSrchEngines

10:18 pm on Apr 21, 2004 (gmt 0)



>Google has stated repeatedly

No they haven't. <snip>

>Having removed the links, they would have to stop using the site descriptions in the SERPs as well. Otherwise they could be in breach of the ODP license.

Bull-oney :-)

Google is using DMOZ sites to filter results. <snip>

"things that used to work don't work anymore" No kidding Shirlock! And...a ton of it don't work at all.

[edited by: Marcia at 5:16 am (utc) on April 23, 2004]
[edit reason] Inappropriate, rude comments removed. [/edit]

cbpayne

11:13 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



Having removed the links, they would have to stop using the site descriptions in the SERPs as well. Otherwise they could be in breach of the ODP license

Where did that come from? Not true.

Genie

11:36 pm on Apr 21, 2004 (gmt 0)

10+ Year Member



I dare say the ODP license wouldn't have been an insuperable problem.

ODP staff did a special deal with Yahoo recently. Yahoo wanted to use ODP site descriptions in its SERPs (for sites not in its own directory). Since the normal on-page attribution would have been impossible, a small acknowledgement elsewhere was agreed instead.

ILuvSrchEngines

12:07 am on Apr 22, 2004 (gmt 0)



Google is not using the DMOZ site descriptions anyway. What Google is now hiding is the directory location of the web sites you see in the top results. This directory location info used to be right below the result of each web page. Many of us noticed however that almost all of the top results just happened to have the DMOZ location below them.

The DMOZ info was a tip-off IMO, that Google probably did not want to be so visible. People can blame it on useabilty or whatever, but it seemed to disappear right after many of us noticed the DMOZ weighting and started talking about it on this forum.

You will also notice that Google implemented an expiring domain penalty which seemed to be targeted at DMOZ listed sites. This was to stop people from buying DMOZ listed expiring domains and using the existing PR (or whatever) to get top placement in Google.

So Google has a way of detecting links made before expiration, which would include a DMOZ link made to the site before it expired. Google would have had to spend a significant amount of time developing and implementing this penalty which would also seem to indicate that Google places a great amount of weight on links, specificlly DMOZ links IMO. Such a great amount of weight that they can not do whatever it is that they do without the expired domain penalty.

People can thumb their noses at me all they want, but in my mind a DMOZ link (if it is current, not before an expiration) is gold for Google, assuming you have not done any really bad sin with your site. I believe that one current DMOZ link would be as good as maybe 500 non-DMOZ links.

The only problem with all this is that most people can not get a DMOZ listing. It's a club that few seem to be invited to.

lloyd

3:45 pm on Apr 22, 2004 (gmt 0)

10+ Year Member



So does DMOZ still carry significant ranking power in Google' algorithm?

Google has just been clever and hidden any reference to DMOZ in their organic listings. But the results are still there.

I tend to agree with ILuvSrchEngines. DMOZ is still a powerful player in the Google algorithm and IMO the strongest. Google aside, DMOZ still dominates the Directory world. You only have to look at all the portals/search engines it supplies results to.

John_Creed

4:06 pm on Apr 22, 2004 (gmt 0)

10+ Year Member



A DMOZ is a good listing and slightly more important than a regular link, but you can definately survive without a DMOZ listing.

steveb

10:25 pm on Apr 22, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



"So does DMOZ still carry significant ranking power in Google' algorithm?"

What the heck does that mean? My site still carries ranking power. All sites do. But "dmoz" isn't some widget. It's hundreds of thousands of pages, all of which carry different ranking influence than each other, in a way no different than the rest of the pages on the Internet.

hutcheson

1:12 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Put the tinfoil hat away, it's blocking your vision.

I've tested your "mom-and-pop" filter hypothesis, and I reported the results here. It was absolutely trivial, given the tiniest random samples, to find multiple small sites (either mom-and-pop or technologically indistinguishable from them) in any of the four quadrants of the In-ODP/Survived-the-Google-changes Karnaugh map.

I conclude that what's causing your perception is over-reliance on an outmoded over-optimization approach.

ILuvSrchEngines

1:26 am on Apr 23, 2004 (gmt 0)



>I conclude that what's causing your perception

Please don't bother to conclude my perception and I won't bother to conclude your perception. But please feel free to conclude your own perception all you want.

ILuvSrchEngines

1:38 am on Apr 23, 2004 (gmt 0)



Not only is DMOZ extremely important to a Google listing, but the location in DMOZ is also a factor in ranking. (call it PR or whatever you want but it certainly favors certain industries with big pockets) For instance, (I can't use real search phrases because or forum rules so you translate) let's say I search for a high value one word phrase.

<widget>

All things being equal for a certain term, all Google first page results are from the DMOZ directory.

<widget lines>

Well into the second and third pages the folling DMOZ listings will start to show.

<widget agents>

So, for the most valuable of all web terms, the one word phrase, a certain DMOZ catagory exclusively holds the entire first page in Google.

Being fair, the same thing happens in Yahoo for one word phrases, but Yahoo uses the Yahoo directory instead of DMOZ.

Yahoo has yet to do a large scale filter effort into 2 and 3 word phrases (as Google has) at this point but it looks like they may be headed down the same insane path as Google.

steveb

1:50 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Actually, as much as you think you are, ILuvSrchEngines I don't think you are disagreeing much with what is said here. You just don't like the reality of it.

A link from Dmoz is no different pagerank-wise than any other link.

There is no "special relationship" that Google gives Dmoz.

That said, Google views Dmoz pages just like they do any other pages on the web. Let's suppose I have an authority site on widgets. A link from me *should* be judged as more of an algorithmic positive than a link from an off-topic site that is not recognized as an authority on any subject. Dmoz pages usually link to quality resources on subjects. Sometimes they blow, and sometimes they should link to more quality sites than they do, but minimally they often link to high quality sites on the topic. A link from such a page should be algorithmically valued, again, just like a link from my widget authority page -- maybe not as much as my widget authority page, but more than an off-topic, non-authority site.

Dmoz links are no different than any other, but that just means that their quality is weighed like any other, and they have a positive value just like a link from my widget site will have to other widget site.

ILuvSrchEngines, I know you are dying for tips from me, but you should just be arguing that Dmoz doesn't deserve being viewed as favorably as it is authority-wise. I'm sure you'll agree that it makes sense for the Dmoz Science page to be looked at more favorably in a science context than some shoe-selling page. It's just the degree that is in question. Most of us think that quality degree should be pretty high. Some folks think it should be lower. But Google is certainly right to be judging all pages on the Internet in terms of their topical quality and judgement.

They look at all pages the same way, but obviously some pages are trusted more than others.

steveb

1:56 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And this isn't just a Dmoz phenomenon with Google. I know of a topic that has a yahoo category, but none exists in Google. The Yahoo category has six listings. Five are in the Google top ten.

Is their listings in the Yahoo category the cause of them being in the top ten? I'd say no.
Is it helpful? Of course.
Google values that Yahoo category, for good reason.

ILuvSrchEngines

2:12 am on Apr 23, 2004 (gmt 0)



>ILuvSrchEngines, I know you are dying for tips from me

No, I am not, and never have been 'dying for tips' from you because you keep showing that you know virually nothing on the topics that you jump in on. Why would I want any info from you? Now on the other hand, if I want tips on how to be a Google cheerleader on virtually EVERY single anti-Google thread then I will come ask you for tips.

ILuvSrchEngines

2:27 am on Apr 23, 2004 (gmt 0)



>Is their listings in the Yahoo category the cause of them being in the top ten? I'd say no.

I'd say yes. If no DMOZ listings are found with that result Google would give top ranking to the next 'expert' (aka directory) page of which Google has defined as the most relevant. This is why many of the characteristics of the Google mom-and-pop/money-word filter seem to indicate that Google is using Hilltop or a form of it. If you are not listed in one of the top ‘expert’ pages (directories) such as DMOZ, Yahoo, LookSmart, etc. you will likely get filtered out completely by the Hilltop like filtering. Google loves directories, but Google is married to DMOZ much the same as Yahoo is married to the Yahoo directory. So Google gives more absolute weight to it's marriage partner DMOZ.

You may be able to get tons of links and brute force your way into the top page of results (without a DMOZ link) for a while, but the Hilltop like algo WILL eventually push you down as it places more directory sites above you in 2, 3 and 4 word money phrases.

steveb

2:55 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Your rants against quality sites are amusing and all, but search engines should like quality. They should like quality, even if you don't.

They aren't perfect about it, but that is why quality Mom and Pop sites continue to do so well, despite the throngs of piffle peddling, low quality, sell-the-same-widget-as-1000-other-sites out there.

BigDave

4:54 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The only thing special about DMOZ is that there are lots of copies of it out there, and several of them continue to count towards PR.

In my industry, I suspect that getting linked from my site makes more of a difference than links from DMOZ. An obscure manufacturer that we had some dealings with shot to the top of some appropriate search terms after we added links, beating out several manufacturers that do have DMOZ links.

But I guess that I'm a Google cheerleader, so I'm an idiot. But then again, I'm the one that is still getting Google traffic.

[edited by: Marcia at 7:54 am (utc) on April 23, 2004]
[edit reason] Formatting. [/edit]

Marcia

5:39 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Getting back to the original topic, which is Google & DMOZ (not *us* or other members), now that Hilltop has been brought into the discussion:

seem to indicate that Google is using Hilltop or a form of it.

Just so we can have something concrete and tangible to measure by, rather than playing Carnac the Mind_Reader, would anyone care to point us to what part of Hilltop specifically makes reference to factors that relate to ODP - or any other directories for that matter?

Marcia

7:39 am on Apr 23, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>Is Google still using the Dmoz listings in the organic results. i.e. the Dmoz description in the Google algorithm.

To answer the original question: it looks like they are not currently showing the description in the SERPs, but we have no way of knowing whether or not it's given any consideration, and if so, how much weight among the 100+ factors used in scoring sites it might possibly carry.

They obviously have access to it, so it might just be one of the criteria taken into consideration when scoring sites for topical relevance for associated keyphrases.

That should about cover it, unless someone has something concrete to add related to the topic or wants to comment on Hilltop and how it relates to this topic.

Hilltop, a Search Engine based on Expert Documents [cs.toronto.edu]

Hmmm.. Krishna Bharat - name sounds familiar. ;)

Meta_Vision

9:02 am on Apr 23, 2004 (gmt 0)

10+ Year Member



(1) Mathematical Impossibility DMOZ/ODP (or any human-edited directory) cannot do what "they promise" to do. There are simply not enough volunteers -- of whatever quality -- for them to be a meaningfully up-to-date categorization of "the whole world wide web." Niche directories are possible. "Whole web" directories, impossible. They cannot help but be full of out-of-date information. There simply can't be enough people involved to handle the growth of information.

Perhaps the illustration of the phone company in the days of human operators would be enlightening. AT&T did a little mathematical projection of how many operators would be needed within a few years ... and that number matched the entire female population of the United States (in those sexist times).

DMOZ/ODP is based on a lie. The lie that the job can be done at all.

Any data to back up that concept? Consider when MSN dropped Looksmart.


QUOTE [DANNY SULLIVAN, Editor Search Engine Watch, 10/7/03] Why drop LookSmart? MSN said testing it did at its MSN Search UK site earlier this year found that dropping the listings increased relevancy.

"The testing was conclusive that the more relevant results were outside the LookSmart listings," [MSN product manager, Karen] Redetzki said. "We're not going to talk publicly about how they [MSN Search UK] measured the relevancy or the methodology, but we did see that the test results showed that the relevancy improved.

BOTTOM LINE: (And, yes, I'll phrase it my way.) Human-edited directories LOWER search quality by being full of out-of-date information. Including dead links. Sites not touched for years. Poor initial description or miscategorization, etc. Google is not stupid .

(2) Yahoo apparently IS that stupid. Yahoo/Overture is, they say, using ODP data. SOURCE OF INFORMATION:
[content.overture.com...]


Yahoo! leverages the Open Directory Project
(ODP) to further improve the user experience
on distribution partner sites. ODP title,
description and category meta data is
used to enhance Yahoo!'s relevant search results.

I HEREBY POSIT THE FOLLOWING "THEORY" BASED ON 1 + 2 {smile}

Google is PHASING OUT the use of DMOZ data
as
Yahoo is PHASING IT IN.

This makes Google (NOW) good. {smile}
and
Yahoo (and all its little search engines, including MSN)
The Evil Search Axis

Later,

MV
(Off to read THE WHOLE HILLTOP ARTICLE
just for fun ... It could shed no light on this {grin})

[edited by: Meta_Vision at 9:06 am (utc) on April 23, 2004]

lloyd

9:05 am on Apr 23, 2004 (gmt 0)

10+ Year Member



I would just like to add this.

Steveb has some valid points, but I see from the sites that I promote that DMOZ listings are not normal links.

Yes a good link is a good link but a DMOZ link is an excellent link. NOT only that, IMO the weight that Google gives to a DMOZ link depends on not the category PR but on the depth of the link. For example the deeper the link the more weight Google gives it.

Why, because it is more concentrated/relevant to the search and has little to do with PR.

jmccormac

11:10 am on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Still using the descriptions from what I can see. It is an updated Dmoz RDF, probably a few weeks old. The main change is that Google moved the link position for its Dmoz mirror from the Google front page.

Regards...jmcc

le_gber

11:13 am on Apr 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I don't think that Google ever used DMOZ description to rank a site.

They use to 'display' the link and description to the Google directory which was based on DMOZ, in their search results.

I am not sure that because they don't display it anymore, that they were or are no longer using it.

Hope this is right

Leo

This 33 message thread spans 2 pages: 33