| This 35 message thread spans 2 pages: 35 (  2 ) > > || |
|DMOZ Editor Login Now Working|
| 4:13 am on Dec 19, 2006 (gmt 0)|
System: The following 2 messages were cut out of thread at: http://www.webmasterworld.com/directories/3171321.htm [webmasterworld.com] by engine - 2:58 pm on Dec. 19, 2006 (utc 0)
UPDATE: December 18, 2006 -- Editors have regained access to the editors server. However, the public ODP scripts (including site suggestion, update listing, editor application, and abuse reporting) are still not available and we currently have no ETA for their return.
Hmm, difficult to post a link if it is changed to some unreadable signs by the forum software.
[edited by: Webwork at 2:41 pm (utc) on Dec. 19, 2006]
[edit reason] Please do not attempt to circumvent the policy reflected in the URL blocking.system [/edit]
| 11:45 am on Dec 19, 2006 (gmt 0)|
at last editing started and lot of editors started their work. Good news
| 11:15 am on Dec 19, 2006 (gmt 0)|
System: The following message was spliced on to this thread from: http://www.webmasterworld.com/directories/3193151.htm [webmasterworld.com] by engine - 2:59 pm on Dec. 19, 2006 (utc 0)
Editors have regained access to the editors server. However, the public ODP scripts (including site suggestion, update listing, editor application, and abuse reporting) are still not available.
| 4:30 pm on Dec 19, 2006 (gmt 0)|
How about backups? Are they working now? But then again, backups may not be a priority, again.
No Backups, Not acceptable, the outage should have never happened. For those that are prepared, there is no storm, let alone a perfect storm. Professionals VS Volunteers, you just do not get the quality help, if you do not pay for it.
Back to Watching.
| 4:42 pm on Dec 19, 2006 (gmt 0)|
good deal. Man that must've been a real nightmare to untangle from what I've read.
| 4:51 pm on Dec 19, 2006 (gmt 0)|
Those who have ever had anything to do with industrial-strength computer operations, will know that backups are ALWAYS working.
It's only the restores that are risky.
And for our techies, the nightmare isn't over. They just let the editors back in as soon as editing functions worked; they didn't wait until the reconstruction/redevelopment work was over.
| 5:21 pm on Dec 19, 2006 (gmt 0)|
For Those who have ever had anything to do with industrial-strength computer operations, we will know that backups are ALWAYS working, because we tested the restores, and left nothing to chance.
Back to Watching,
| 6:09 pm on Dec 19, 2006 (gmt 0)|
Ah, the unshakeable omniscience of inexperience.
| 9:30 pm on Dec 19, 2006 (gmt 0)|
Ah, the unshakeable omniscience and inexperience of an ODP Editor.
The biggest thing I cannot stand about the ODP is the "better than thou" attitude of the more outspoken editors.
I am experienced, 15 years of being a network engineer, and I never lost any data, for companies that have databases that make the ODP look like a small bag of peanuts. We did backups, tested the restores, created disaster plans, & tested them offsite by rebuilding the entire environment on new boxes, all for testing purposes to ensure that there was never a problem. That would be the AS400, Unix, Windows servers, mail servers, Metaframe server farms with load balancing(one server per each farm for the rebuild test) & everything these companies relied on to do business. This would include, but not limited to, two of the largest airlines, two of the larger phone companies, and several other fortune 500 companies.
I would be considered one of the more experienced in the industry, and was paid well for these designs & services, and I never failed. I am not an amateur, a volunteer who does not have a clue on backing up databases. (as demonstrated by the people in charge of the backups, who if they were employed, would have been fired for not doing their job)
And U Mr ODP Editor?
Back to Watching
Edited to add, I guess you are right, I have never experienced a disaster I could not recover from in hours, so I am guilty of this inexperience; I have never experienced an outage of this size and duration. I find it almost incomprehensible.
[edited by: WW_Watcher at 9:53 pm (utc) on Dec. 19, 2006]
| 9:52 pm on Dec 19, 2006 (gmt 0)|
Just to point out that the volunteer editors don't run the ODP servers and don't have the permisssions to perform any ops duties. The servers are provided and maintained by AOL professionals in one of their data centres.
| 10:09 pm on Dec 19, 2006 (gmt 0)|
My experience has mostly been with the actual computer manufacturers, on the development side, where it is by no means so trivial to identify potential single points of failure. And yes, I have experienced failures where the techniques you describe would have been either impractical or inadequate.
| 10:25 pm on Dec 19, 2006 (gmt 0)|
The boxes are AOL's, The tape/media is AOL's, the professionals who are in charge of them from AOL(and there is a debate if there are any of those), are responsible for working with the ODP Programmers, DBAs, & SysAdmins to get a a list of what to backup, and when, and then ripping it off to another media. This requires a collaboration of people to accomplish this task, so that the applications & data are restorable.
All large databases are a pain to backup, and restore, but this is not new news, and everyone who values their systems have worked out processes to cure these issues. It is not enough to rely totally on raid arrays & multiple boxes for hardware failures, you have to have good backups, you have to test your backups, or face what the ODP is facing now. This is all I am saying. If AOL/ODP have not fixed the backup/restore issues, it can happen again anytime.
If I was still in my previous profession, I would be using the current problems of AOL/ODP as examples to my clients to justify the cost of preparation for any disaster.
I place no blame for the current ODP problems on the editors. From my perspective, You/They are just the users of an application, nothing more.
Back to Watching
Edit to add,
I also consider the editors the only people who suffered during the outage. It made no difference to the DMOZ, other than being another strike to its reputatation.
[edited by: WW_Watcher at 10:32 pm (utc) on Dec. 19, 2006]
| 10:50 pm on Dec 19, 2006 (gmt 0)|
Perhaps I didn't make myself clear. All of those functions on the editor, forum and public facing servers are undertaken by professional AOL staff. Other than as application program users, editors have no access and most have precious little idea how the systems work.
|..working with the ODP Programmers, DBAs, & SysAdmins |
| 11:23 pm on Dec 19, 2006 (gmt 0)|
Sorry jimnoble, I guess I did not understand.
Am I to understand that the DMOZ is an application, running on a couple of boxes at AOL, and AOL is totally responsible for the Development (or lack thereof) of the application, all of the system & database administration, backups & procedures for recovery, and the editors are pretty much clueless on how any of it works, & are just a private club of users, and AOL is in charge of everything and totally to blame for the outage?
If that is the case, my bad! No chance it will hurt AOL’s rep, it cannot get much worse.
(either way, if the backup/restore issue has not been resolved, it will happen again, not a question of if, a question of when)
Back to watching (I already spent more time thinking about this thread, then I did when I submitted my site to the project years ago)
| 6:42 am on Dec 20, 2006 (gmt 0)|
The tone of discussion seems be sliding downhill slightly :-(
With reference to
|backups may not be a priority |
then I think we all know that DMOZ wasn't the first organisation to find out that backups had been neglected, and it won't be the last.
It's easy to say "<foo> should never have happened" after it has happened, perhaps we should avoid this kind of comment.
With reference to
|DMOZ is an application, running on a couple of boxes at AOL, and AOL is totally responsible for the Development (...) the system & database administration, backups (...) and the editors are pretty much clueless |
then I think that's a fairly good description of the situation. Editors see DMOZ as a web application. I'd bet you a beer that most editors have no idea how DMOZ works internally, where it's hosted, or who's actually responsible. In fact, since most DMOZ editors aren't necessarily I.T. 'Pros' (for want of a better phrase) I think 99.9% of editors have no idea what a data centre really looks like, and have certainly never been in one.
With reference to
|For those that are prepared, there is no storm, let alone a perfect storm |
How many businesses are there who can truthfully say there is zero chance they could be hit by a combination of events like the ones that hit DMOZ?
Webmasterworld was hit by exactly this kind of "multiple event" disaster not so very long ago, and was offline for several days as a result, without even a holding page, AND the recovery took quite a while once the site was back up.
I think we all assumed that Brett knew what he was doing (and most of us *still* assume that he knows what he's doing). WW_Watcher, could you give us your analysis of the problems at Webmasterworld? ;-)
| 1:00 pm on Dec 20, 2006 (gmt 0)|
|How many businesses are there who can truthfully say there is zero chance they could be hit by a combination of events like the ones that hit DMOZ? |
Short of a global nuclear strike, put us down for a zero chance.
Disaster planning is not rocket science! (trust me im a systems analyst working on an astrophysics degree).
Blame for this extended downtime either falls on Systems operations (if they are at fault), or AOL management, if they did not give resources and guidance to Sys Ops.
One way or another, their should be some new job vacancies in either IT or IT management at AOL :)
| 1:12 pm on Dec 20, 2006 (gmt 0)|
|Short of a global nuclear strike, put us down for a zero chance [...of a series of mistakes] |
Really? :-) Your Operations staff don't make mistakes? Where **DID** you find them?
I've been advertising for StaffWhoNeverMakeMistakes(tm) but somehow I only get humans applying for jobs. What's the secret?
| 1:50 pm on Dec 20, 2006 (gmt 0)|
heh, I have a DMOZ backup if they want it :)
| 1:53 pm on Dec 20, 2006 (gmt 0)|
StaffWhoNeverMakeMistakes(tm) usually achieved that exalted status by never doing any work and by never making any decisions. Why would you want to employ people like that :)?
| 1:53 pm on Dec 20, 2006 (gmt 0)|
Thanks, you remind me to activate my account.
| 2:40 pm on Dec 20, 2006 (gmt 0)|
If “backups may not be a priority”, then whoever had that, must have considered it not worth recovering if it was lost. I kinda agree with that statement with the ODP, if it made me nothing, and cost me, and everyone considered it not a priority, then why bother.
“should have never happened”, If we do not learn from the past, we are doomed to repeat it. Many of us have seen disasters, and prepared so that it did not happen to our environments.
I never said there was a zero chance, I said there was no storm, or perfect storm. We restored stuff daily; Users delete stuff, lose stuff, change stuff & then regret it. Hardware fails, database corruption, This is ALL common occurances, and happen daily around the world, and for those who have prepared, no big deal. I documented & trained people to follow procedures, with different procedures for backups & restores for different applications. It was not flawless, but my clients were not exposed to massive data loss, or down time. (Would it be acceptable if your hospital lost your records & you got the wrong meds, Would it be acceptable if the airlines lost their routes & passengers, Would it be acceptable if a phone company lost all of it’s customers in a data loss? I say no, and most of these businesses have prepared, so that it is not a concern.
I was not privy to the internal problems of WW, I was not aware that he had a problem that he had to use a backup to recover from. The question should be, did Brett learn from this experience, and is he getting good backups now? Is he better prepared now, so that it does not occur again? If not, he is also just biding his time until the next disaster. Brett, please do not let this happen, I would miss WW, unlike the ODP, which I really do not care if it returns or not.
Back to Watching,
| 3:24 pm on Dec 20, 2006 (gmt 0)|
|If "backups may not be a priority", then whoever had that, must have considered it not worth recovering if it was lost |
I quoted the phrase "backups may not be a priority" directly from your earlier post. Was that simply your conjecture or do you have inside info from AOL?
|I kinda agree with that statement with the ODP, if it made me nothing, and cost me, and everyone considered it not a priority, then why bother |
If everyone hates the ODP so much, and it's so useless, then surely all we have to do is ignore it and/or build a replacement. As far as I recall, Google gained users from Altavista, and Altavista essentially died off, because Google was simply better, not because thousands of bloggers spent their time posting "Altavista is dead!".
Why do we all spend so much time discussing ODP/DMOZ if it's already dead-and-buried?
|The question should be, did Brett learn from this experience, and is he getting good backups now? |
Agreed. Based in part on your comments, I think the key point for those in charge of DMOZ is not "should we pull the plug on this project" but instead "did we learn from this experience and are we getting good backups now".
Good hindsight is really easy (and often unhelpful).
Good foresight is substantially harder.
| 5:05 pm on Dec 20, 2006 (gmt 0)|
Webdoctor, I do not hate the ODP, not sure where you got that, I just do not really care enough about it to hate it, it just does not matter anymore. I would happily buy it, & use it to boost the PR & traffic on other sites, I do not consider it having much value beyond that, & I would not support it for free.
My point is that if organizations do not value their data & applications enough to do disaster planning, follow proper backup procedures & test the restores to see if the disaster planning worked, then they deserve to lose their data & applications.
I totally agree with your hindsight & foresight statements, especially on the foresight, that is exactly what I have been preaching, it requires work, not one time work, but continual monitoring of the success of the backup, & test restores to ensure that you are getting it all, & in a fashion that can be restored. It is a never-ending job that professionals do every day to protect organizations from data loss.
Enough wasted effort on this subject, it's not even my problem,
Back to Watching (what I normally do, not posting)
| 5:39 pm on Dec 20, 2006 (gmt 0)|
Can i ask you guys who know, was there an extraordinary issue preventing the practice of continous backing up, as practiced by everyone else?
Was it too costly, how come, i've seen terabybte NAS backup devices for under $1,000 , how many could you need?
Okay, I have zero expertise with enterprise class solutions, but reading ww_watchers comments suggests the processes involved are not exactly rocket science , relative to me
Anyway, an answer would be helpfull as it would inform more about the issues faced by larger networks
| 6:10 pm on Dec 20, 2006 (gmt 0)|
Ok, One more post, because Vite_rts asked,
Vite_rts, I was not the poster who is the Rocket Scientist In Process, That would be Lgn1. But I do think the two of us agree on the issue of backups & restores.
Most backups of data & applications are relatively simple, just rip it all off to another media & restore it if need be, but when it comes to high availability, live database applications & the continual locking of records for transactions, sometimes it makes it more difficult to get a complete restorable backup. More difficult yes, but never impossible. Many of the major database vendors have worked with the backup companies to develop solutions for their applications & databases to cure these issues & get restorable backups, I do not pretend to know what the ODP runs on, or how it works, so I would not/can not offer any solution. Sometimes you have to think out of the box to get the backup in a fashion so it can be used to recover from a disaster. The backup solution depends on the database software & the application & the way it is used, no one shoe fits all. But there is always a way, if the endeavor is worth protecting.
Ok, I really am trying to be done on this thread,
Back to Watching
| 2:41 pm on Dec 21, 2006 (gmt 0)|
I never said people never make mistakes. In fact, your disaster recovery plan, must include the fact that people will make mistakes, and plenty of them; especially if they are under extreme presure.
Backups are great for recovering from those small disasters.
Offsite archivals (and im talking multiple copies), are going to protect you from those days, when you think the world is going to end.
Also of note; you still can't become a DMOZ editor, or suggest a site, as this service is still unavailable, so DMOZ is not totally back.
The good thing, the DMOZ editors won't have any spam to contend with for a while, so they may actually get caught up in reviewing all the suggested sites in their inbox.
If this service is down for long enough, their may be a window of opportunity to get your site reviewed in a timely manner, when (if) the suggestion tool becomes active again.
| 6:23 pm on Dec 21, 2006 (gmt 0)|
>If this service is down for long enough, their may be a window of opportunity to get your site reviewed in a timely manner, when (if) the suggestion tool becomes active again.
There are a wealth of misconceptions here. First, even at this point it is absolutely certain that many site suggestions were not lost. And second, with time off to relax and rethink their activity, many editors will come back with pent-up priorities of their own. It is far more likely that, for awhile, the always-low priority generally given site suggestions will be especially evident (since there are so many high-priority activities to tempt the volunteers.) I'd expect renewed focus on the topics that are likely to interest volunteers most; on quality-focussed re-reviews of existing sites; continued data recovery and data validation efforts; some distraction on discussions of future directions, and so on.
| 7:42 pm on Dec 22, 2006 (gmt 0)|
|This would include, but not limited to, two of the largest airlines, two of the larger phone companies, and several other fortune 500 companies. |
Sounds like you had the good fortune to be working for people who actually placed a reasonable dollar value on disaster planning. You probably know other sys admins who had all the necessary smarts, but could not convince management to actually pay what it costs to implement a serious disaster recovery plan.
Like a foster child kicked from one home to the next, ODP is never likely to get that kind of loving care.
| 10:06 pm on Dec 23, 2006 (gmt 0)|
Gee, and I thought AOL was a Fortune 500 company :)
And I thought they called it America offline, for its slow dial up service in the mid to late 90's.
| 12:31 am on Jan 2, 2007 (gmt 0)|
|It is far more likely that, for awhile, the always-low priority generally given site suggestions will be especially evident (since there are so many high-priority activities to tempt the volunteers.) I'd expect renewed focus on the topics that are likely to interest volunteers most; on quality-focussed re-reviews of existing sites |
How naive. Probably far more likely that the majority of the editors who are casual editors will just throw in the towel and never edit again.
Sure, some of the diehards may be reinvigorated. But the number of active editors is sure to drastically drop.
The diehards can't come close to handling things as is with the casual editors' assistance. The ODP will just fall further behind as a result of this very extended outage.
| This 35 message thread spans 2 pages: 35 (  2 ) > > |