So why is ODP so slow?

Forum Moderators: open

Message Too Old, No Replies

So why is ODP so slow?

Jon_King

1:50 pm on Jan 21, 2003 (gmt 0)

I believe I read something about hardware changes at ODP. Is this the reason for continued outages and slow performance?

rogerd

2:01 pm on Jan 21, 2003 (gmt 0)

The problems over the last few months seem to be related to a combination of upgrade processes and extremely limited technical staff. The RDF dump (used by some sites that feature ODP listings) is months behind right now.

Dumpy

2:28 pm on Jan 21, 2003 (gmt 0)

It is interesting to follow the progression of companies. Netscape was originally developed over a weekend. Look what they are able to do now with a problem that could be solved in a weekend with free software.

I was experimenting with some software, testing it on a category and my IP was banned.

The software worked perfectly and I no longer have to rely on the rdf dump.

I predict there are many hundreds working on such solutions.

windharp

2:56 pm on Jan 21, 2003 (gmt 0)

You dont have to work on these tools - a lot of cgi based live feed routines already exist.

To come back to the question:
"Why is ODP so slow" - mainly because servers are suffering from to much load. You see the point when some RDF users are switching to live feed?

DMOZ servers where not intended for live feed - if you want to have faster access for cgi based solutions, you could use the mirror at ch.dmoz.org - if the synch process is running smoothly it should be about one day behind the main site.

Dumpy

3:20 pm on Jan 21, 2003 (gmt 0)

I don't use live feed solutions. I use static pages, generated from parsing the rdf dump. The solution for a work around is to create static pages, using my template, from the OPD database. So, I need to access only when I decide to update, and then only pages that are changed.

The mirror site has a lot of "not found on this server" pages....it seems to lag considerable in being updated. However, it is fast.

I would predict this solution will be available shortly for sale. And, as OPD bans live feed sites that get to a size that impact their server, this solution will prevail. It offers distinct advantages of freshness over RDF Dumps, and the little site can become competitive over the BIG site.

The software I use is free off the shelf stuff and I would certainly buy a version more user friendly.

rmjvol

9:19 pm on Jan 21, 2003 (gmt 0)

The Question: So why is ODP so slow?

The Answer: Infrastructure [rpfuller.com]

hutcheson

10:21 pm on Jan 21, 2003 (gmt 0)

There are two obvious causes for slowdown: server load (heavy editing, and malicious non-editor activity. If you calculate about 14000 edits per day (adds, deletes, moves, etc.) and about 7 page views per edit, then editors would be doing about 100,000 page views daily. One sick spider trying to grab all 330,000 publicly-viewable category pages -- well, you do the math. And it's not just one sicko out there.

If I correctly understand what has been said, then another cause of the _current_ slowdown is heavy _development_ work, making basic changes to the system for the sake of scalability (I think this can only mean distributing editing work across multiple servers, since searches and forums are already on different servers.)

Programmers will immediately recognize the significance of this: not only does it represent a MAJOR investment in programming effort, it is the sort of investment that is only worthwhile in consideration of the long term.

rogerd

10:35 pm on Jan 21, 2003 (gmt 0)

Nice link, rmjvol! Hilarious. And scary, too, since it appears that it was posted almost a full year ago!

One ray of hope - now that they seem to be down to one tech, that should free up the Quake platform for other uses... ;)

hutcheson

2:26 am on Jan 22, 2003 (gmt 0)

Hey! that Quake box belongs to the _metas_!

HoloC76

3:40 pm on Jan 22, 2003 (gmt 0)

Well, they're working on the forums, anyway. My dashboard says they'll be down for 24 hours or so starting this afternoon - moving them to a new machine.

victor

4:35 pm on Jan 22, 2003 (gmt 0)

Juts to clarify, "they'll be down for 24 hours" -- "they" is the forums, not the whole ODP. At least that's how I read the status message.

With the forums down, editors won't be able to talk to each other or comment on the ODP's recent slowness. Maybe we'll have to squeeze in some editing instead :)

HoloC76

4:54 pm on Jan 22, 2003 (gmt 0)

Ooops :o - yes, the FORUMS will be down for 24 hours. Sorry to confuse anyone. They suggest editors use the public forums during that time, if they need to.