homepage Welcome to WebmasterWorld Guest from 54.163.139.36
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Directories
Forum Library, Charter, Moderators: Webwork & skibum

Directories Forum

    
Why Sites Disappear from Dmoz (Open Directory Project)
brycen

10+ Year Member



 
Msg#: 4638805 posted 9:51 pm on Jan 20, 2014 (gmt 0)

After reading through various dmoz threads:
[webmasterworld.com...]
[webmasterworld.com...]
[webmasterworld.com...]

I see one notable topic not covered: why sites my disappear from the directory.

It's not always editors that do it. You see the core of dmoz languished for decades. Various editors built tools external to dmoz, one of which is a quality check which gets run at somewhat inconsistent intervals.

Sites that are down for two scans in a row are removed from the active directory. It can take months, years or decades for an editor to notice and fix the problem. The goal is if the site was down, perhaps it needs to be revisited by a human.

I've rescued dozens of sites from this purgatory, and seen perfectly functioning sites that sat unlisted for two...three...five years.

 

Webwork

WebmasterWorld Administrator webwork us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4638805 posted 11:16 pm on Jan 27, 2014 (gmt 0)

Thanks for the insight brycen.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4638805 posted 5:44 pm on Feb 3, 2014 (gmt 0)

I understand why they did that because if the site transitioned to a domain park or a different owner then you wouldn't want to reinstate it automatically.

However, retaining a simple hash code that represents the content of the original indexed page could be enough to let it automatically be reinstated if the hash code matches when it comes back online.

Oh well, they'll probably never fix it now, so many years later.

I'm not sure anyone really uses it anymore as it's so Web 1.0. ;)

Would it kill them to at least make it a RWD site (Twitter Bootstrap anyone?) so it's at least mobile friendly at relevant to this decade?

blend27

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4638805 posted 7:43 pm on Feb 3, 2014 (gmt 0)

I used DMOZ quiet heavily to get ips of the sites hosted were they were, on to get the IP ranges of Hosting companies that host those sites to block the ranges. That was about it.

DMOZ Bot comes in and gets 403 for a while now for one of the sites listed there. No human Traffic from that listing except occasional link builder, and guess what; from India...

I remember when we got the listing for this site(8 years ago or so). I was very proud, really proud. I thought we were recognized by... Now days, the only way in my book, DMOZ, is utilized is for DUMPS of data, scape source etc.. Someone dumped the data on 400 sites that got indexed about 2 years ago, nasty stuff when it come 2 Google..

... Currently 2 sites in our category page are KAPUT, several month now. Landing Page from the Host.


Recently:
IP: 65.44.192.153
UA: dmoz_scraper/1.0

I could go on but..,

@brycen, I appreciate you taking the time to share you knowledge.

brycen

10+ Year Member



 
Msg#: 4638805 posted 1:18 am on Feb 4, 2014 (gmt 0)

Thus: a site that gets de-listed by RoboZilla (the dmoz quality check tool), effectively gets treated as a brand new listing.

Note that AOL recently turned over significant amount of the dmoz code over to the community, and if you look at the dmoz blogs you'll see announcements of new software releases. The dmoz editor community, however, is very similar to that of prior years and decades: there is a small core of senior dmoz category and meta moderators who exert a high degree of control, often overriding edits of less senior contributors.

While 89,255 editors are listed on the dmoz home page, the number of active editors is considerably smaller. Detailed statistics on dmoz internals are available to dmoz editors, but rules prohibit publication or sharing of the results. The public RDF dumps however are the source of much analysis, and could be analyzed by anyone interested.

Jim_Berry



 
Msg#: 4638805 posted 5:40 am on Feb 4, 2014 (gmt 0)

Thanks for your notable insights.

I would hate to see DMOZ go down.

super70s

5+ Year Member



 
Msg#: 4638805 posted 6:29 am on Feb 4, 2014 (gmt 0)

While 89,255 editors are listed on the dmoz home page, the number of active editors is considerably smaller.


Yeah, they need to require all dmoz editors to prove they're still breathing, or at least answer an email every 6 months or something.

fathom

WebmasterWorld Senior Member fathom us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4638805 posted 3:11 pm on Feb 4, 2014 (gmt 0)

Yeah, they need to require all dmoz editors to prove they're still breathing, or at least answer an email every 6 months or something.


Your privileges expire after four months of inactivity.

super70s

5+ Year Member



 
Msg#: 4638805 posted 1:38 am on Feb 5, 2014 (gmt 0)

Your privileges expire after four months of inactivity.


Glad to hear that, I finally got a site listed on there after several years of submitting about once a year. I think it's in a deeper sub-category than the one I always aimed for though. I just assumed the editor was MIA.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Directories
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved