Using Dmoz Data with ASP Scripting

[dhaliwal]

i am using dmoz data on my website and am using some code from a free script to do this.

I am currently using asp script which makes use of xhttp to get the data from dmoz. It uses MSXML3.dll which goes on error if there are many people on the website.

Now even if there are 10 persons on pages frmo dmoz, my whole site goes on error,

Can anyone suggest some good code in ASP, or in PHP that can help me.
I hope someone would have made his own code and would like to share, i have already searched a lot on hotscripts and other places.

Thanks for your time
Dhaliwal

[dhaliwal]

will anyone advice anything on that front please

[g1smd]

Sounds like you scrape the directory pages directly. Maybe you should look to download the RDF dump from [rdf.dmoz.org...] and run your own local copy of the directory instead.

A new RDF file appears almost every week, and recent work now sees that the whole directory is UTF-8 encoded (yeah, all 2.2GB of it).

[dhaliwal]

2.2 GB

I don't think any database system will support such a big amount of data.

Now my problem is that
i don't want a small part of the dmoz, like just the tech directory, i want to show it in full

If i use the direct scraping from dmoz using the many scripts available in market, it cause troubles with asp and this trouble comes when there is more traffic as the XML dll goes on error.

Hope someone has some good advice on what script to be used to have the better integration of dmoz directory on my site.

[dataguy]

I've had to switch from XMLHTTP to ASPTear on a few of my ASP sites. ASPTear is much more stable and offers a few more features for scraping, though I gave up scaping the DMOZ years ago.

There is a free version with a smaller feature-set, but it's well worth the registration fee.

[pete_m]

I don't think any database system will support such a big amount of data.

Actually, any decent DB shouldn't have a problem with 2.2GB of data.

If you're using mySQL, you could pack it into 1 table and use

myisampack

to compress it (as long as you don't mind the data being read-only).