homepage Welcome to WebmasterWorld Guest from 54.166.173.147
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Directories
Forum Library, Charter, Moderators: Webwork & skibum

Directories Forum

This 40 message thread spans 2 pages: 40 ( [1] 2 > >     
Building your own directory
what are the options?
fom2001uk




msg:485517
 8:51 am on Oct 18, 2002 (gmt 0)

There are lots of directories on the Web and I've been wondering how easy it is to set one up. What are the options for building your own directory?

What kind of software do you need?
Can you build one from scratch yourself?
What are the advantages of cloning an existing directory (Dmoz, etc)?

I'm really just starting to think about it, so any useful pointers would be helpful.

 

Damian




msg:485518
 9:00 am on Oct 18, 2002 (gmt 0)

In this thread [webmasterworld.com] you'll find some recommendations for off the shelf solutions.

Writing your own can be a lot of work!

The advantages/ disadvantages of cloning dmoz will depend on what it is you're trying to do. I think engines like Google may automatically ignore you for being a duplicate of another site.

mack




msg:485519
 9:01 am on Oct 18, 2002 (gmt 0)

From the offset building a directory isn't as hard as you may imagine.

First thing to think about is how large do you ant your directory to be? If your directory is going to be pretty small you could build it using static pages interlinked. If you are going to list a lot of sites then you really should think about using software for directory management. One of the more popular software options available is gossamer threads link 2.

You then need to build your category structure, this can take some time. You need to enable categories that can later be broken down into sub categories ect.

The entire process really depends on what sort of directory you are going to build, general or specific. You where also asking about incorporating other directory feeds into your site such as ODP. Importing ODP date is good in a way because it gives you a great directory full of listings with little work on your part, then you have the downside, why use your site when the user could get it else where.

It is always worth allowing users to add sites. That was you aren’t having to find sites, you can let your users and other site owners make suggestions.

Quadrille




msg:485520
 9:19 am on Oct 18, 2002 (gmt 0)

I think engines like Google may automatically ignore you for being a duplicate of another site

You'd think so, but Google doesn't - search for something unusual, and often the first ten results are ODP clones; Search for a (recently) closed site, and you'll get pages and pages of them!

You need to think carefully about what you hope to get out of it, and, whether it's fame, cash or simply satisfaction, I'd suggest going for a niche; in a small area, even starting with the ODP dump, you can soon create a useful and up-to-date directory, with the possibility of ads / sponsorship / affiliations; you'll also get plenty of visitors to pass on to any commercial sites you might have.

Good Luck with it ...

cornwall




msg:485521
 9:28 am on Oct 18, 2002 (gmt 0)

There are lots of directories on the Web and I've been wondering how easy it is to set one up

I am not sure why you are thinking of setting a directory up. Large directories that try to cover the whole web are already there in Yahoo and DMOZ (and struggling to keep up).

New arrivals like JoeAnt and GoGuides have (at the risk of annoying editors there) very small data bases for general directories. Both of them claim 100 plus editors and have been in "business" for a year and a half.

My feeling would be that it would only be if you set up a specialist directory that you could hope both to index the subject fully and at the same time attract traffic who were prepared to come in numbers for the specialist content offered.

engine




msg:485522
 9:45 am on Oct 18, 2002 (gmt 0)

We had a nice discussion on this over in the UK forum. [webmasterworld.com]

fom2001uk




msg:485523
 11:13 am on Oct 18, 2002 (gmt 0)

Thanks for the responses. Never realised this was so topical !

To answer a couple of questions, I'm thinking niche directory or hub, possibly regional, or maybe B2B/ B2C.

I was also thinking of geeting the bulk of the data from other directories (probably Dmoz mostly). I also like the idea of getting people to submit suggestions.

Why am I doing it? Not for fame or fortune, but purely for marketing leverage.

Craig_F




msg:485524
 11:36 am on Oct 18, 2002 (gmt 0)

I've been thinking about this too, but more on the side of turning a profit with paid listings.

It would be B2B and the listings themselves would be affordable to the market I'd approach since they are accustomed to paying heavy fees for inclusion in offline buyers guides and the like.

How viable is this profit wise? Anyone have any experience? I know that I've been working on building links to a site that would fit in a directory like this and would easily drop $20-$100 for a yearly listing if the directory was focused and well linked itself, but would enough others be willing to do so for it to be profitable?

Abrexa_UK




msg:485525
 12:45 pm on Oct 18, 2002 (gmt 0)

From my experience of having done this, you won't make anything from submissions. Our directory was built using exactly the methods discussed here: we took a regional DMOZ dump and modified it to set up initially. Then we started adding sites.

The number of submissions that you will get depends entirely upon whether you charge or not. If you charge anything at all, you will get only a handful of submissions (relatively). We charge as small amount as possible to cover the editor's time in adding sites, yet still get less submissions than we would like.

We did try free submission for a while and had the opposite problem; too many submissions. We were just swamped in a matter of days with endless junk sites.

Fortunately, we are now making enough from the advertising to cover some free submissions for non-commercial sites, which we will introduce in the next few days.

You need to be prepared for the reviewing time as well. It can take anything from 5 to 15 minutes to review a site if you do it properly. Often the submitted information needs to be completely rewritten, or you will need to request more info. Are you going to check for browser compatibility? Secure servers? Privacy policies? The more facets you look for the longer the review will take.

The other issue of course is whether your listing will provide value for money. People investing $ in a listing want to know that they are likely to get some traffic in return. Hence your site will need to have a good amount of traffic, which of course takes time to build. Our site has only really been receiving large numbers of visitors for the last 4 months or so. Prior to that we were struggling to get enough visitors to justify a listing fee.

To sum up:
- you won't make any money from the submissions.
- you need to allow a lot of time for reviewing purposes (at least 5 minutes per site).
- you need a lot of visitors to make it work.

Craig_F




msg:485526
 1:14 pm on Oct 18, 2002 (gmt 0)

thanks for the response. are you referring to the directory in your profile? I ask because while you've obviously done a good job of it, it's a general directory and in that case I'd think you would have much more difficulty making money from submissions.

but the directory I'm thinking of would be tightly focused on a specific industry, which I think changes things quite a bit.

for example, with a general directory I will never pay much to submit unless there's some serious traffic or link pop associated with the link, but with a tightly focused directory I can see paying $50 or so for the link pop and the targeted traffic -- even if it's not a lot of traffic in comparison to a general directory listing.

To me as a site owner $50 a year for a solid very industry related link is a no brainer. To me as the directory owner I'll gladly review and edit 3 submissions an hour for $150.

But, I know it's not that easy/simple. Comment away, I'm very interested to hear why this will/won't work.

red_bull




msg:485527
 1:18 pm on Oct 18, 2002 (gmt 0)

I know someone who runs a web directory.

I recently helped out with reveiwing (add/remove) from the submissions.

One thing I was advised to watchout for was the Google status of the site being linked to. If the pr=o then think, is this site new or is it penalised. If you suspect it has been penalised a link to it is way to risky.

Mike_Mackin




msg:485528
 1:31 pm on Oct 18, 2002 (gmt 0)

Now THAT is an interesting editorial policy :(

The Contractor




msg:485529
 1:37 pm on Oct 18, 2002 (gmt 0)

One thing I was advised to watchout for was the Google status of the site being linked to. If the pr=o then think, is this site new or is it penalised. If you suspect it has been penalised a link to it is way to risky.

I think if you are running a directory of any sort your primary focus should be to provide visitors to the sites in your directory. I wouldn't even consider thinking of Googles PR on a site as a editorial policy.

My 2-cents

Brad




msg:485530
 1:57 pm on Oct 18, 2002 (gmt 0)

Craig_F,

I think before you can charge for submissions you will neeed to be a proven source of traffic.

As far as directory scripts go I advise going for something that is completely spiderable like a perl script that "builds". There are some PHP/mySQL scripts available however I am not sure how spiderable they are. Perhaps somebody here who knows about PHP can answer that more concretely.

>>One thing I was advised to watchout for was the Google status of the site being linked to. If the pr=o then think, is this site new or is it penalised. If you suspect it has been penalised a link to it is way to risky.

I hate to say it, but for a little guy starting out it is something that you probably have to consider. I hate feeling coerced into using M$ IE browser to review sites. Sad but true for those starting out. I think once you are well established you might get away from using it.

cornwall




msg:485531
 2:10 pm on Oct 18, 2002 (gmt 0)

but the directory I'm thinking of would be tightly focused on a specific industry, which I think changes things quite a bit.

with a tightly focused directory I can see paying $50 or so for the link pop and the targeted traffic

I am with you all the way on those sentiments.

Specific sites that I have had indexed by "small" general directories like JoeAnt, GoGuides or Abrexa_UK produce very little traffic. However I have found that industry specific mini portal sites I am developing, but not promoting (to the extent that I am wanting to build traffic before I actively sell inclusions) are already attracting interested individuals who are prepared to pay for inclusion (or the opportunity to alter the directory entry ;) )

In other words the targetted traffic is already showing up in people's logs and that is driving them to investigate, and indeed be prepared to pay sums larger than the $50 you mention

red_bull




msg:485532
 2:15 pm on Oct 18, 2002 (gmt 0)

It is not really an editorial policy, if a site is of high quality and has unique content that would be of use, I would list it. But if it is PR0 then you just take a look as to why. If it is blatent spamm then it is not usefull to the end user and could lead to PR0 being passed back to the directry page.

The Contractor




msg:485533
 2:18 pm on Oct 18, 2002 (gmt 0)

I have to disagree with red_bull and Brad :(

If you are going to have a directory you need to do the following:

1.Write and solicit content to attract visitors and search engines. Without visitors or search engines you will go nowhere. When site owners see you are coming up in the results for their keywords you will attract listings.

2.Review the sites and see if they will benefit “visitors” as this is who you are trying to attract – getting listings will follow automatically. If you keep the visitors coming you will provide a benefit for the submitters to your directory – without the visitors your submitters may as well write their URL on a barren sidewalk. Do everything to bring traffic to your site that you would with any other site – this will benefit the submitters.

3.Do “Not” use a add url script or you will be overwhelmed with spam. Charge a small fee or you will be overwhelmed with spam. Finally – look at it as a long term project for years to come and don’t get hung up on instant success.

My 2-cents

red_bull




msg:485534
 2:32 pm on Oct 18, 2002 (gmt 0)

Is certainly is a long term project. You are talking in terms of about three years before you will start getting decent page views. I think you should use an add url script because this makes it easier for a visitor to add a site and also makes editing a lot easier. I dont think any small scale directoried can justify the cost of PFI, granted it takes time to add an entry , but anyone who is willing to pay will expect a descent return on the cost.

Some people have said it is not right to not link to PR0 websites, simple question , would you link to them?

Brad




msg:485535
 2:35 pm on Oct 18, 2002 (gmt 0)

Depending on your topic you may want to think about syndicating your search results -- for a fee.

Depends on your niche.

rogerd




msg:485536
 2:54 pm on Oct 18, 2002 (gmt 0)

>>syndicating your search results -- for a fee<<

Or for favorable linkage...

I like the idea of a topical directory on a site. It's not difficult to get far more depth than Yahoo & Dmoz, particularly if you stick to a narrow field of interest and do your own spidering. It's a great way to add useful content, get inbound links from some sites, and establish a dialog with other sites in your area of interest. Be wary of your tools, of course - Zeus software made some parts of creating a link directory a lot more efficient, but proved to be risky in terms of Google penalties.

cornwall




msg:485537
 5:17 pm on Oct 18, 2002 (gmt 0)

Is certainly is a long term project. You are talking in terms of about three years before you will start getting decent page views.

You are in the right neck of the wood there with that timing. On one of my development sites - a directory site for a niche market, the first year brought little in the way of click throughts to listed sites. It then took off quite steeply and after 20 months seems to be starting to deliver reasonable traffic to a niche market. Happily I do enough individual sites within the niche market to measure progress on referrals.

My guess is after two years, then most listed sites will see appreciable traffic, which by its nature is highly targeted. It is only then that I believe there is something to go out and really "sell"

Marcos




msg:485538
 6:46 pm on Oct 18, 2002 (gmt 0)

Just for the record, we have put more than 800 man hours to set up the software infrastructure of a ODP based directory, and we are not even close to finish it.

Parsing and updating odp data is not that dificult, search engine optimization takes time, but is not a programatic intensive issue, is more like play a few minutes with the algo, launch the searching/indexing engine, and wait a few hours to see who good it is. But setting up a good editor/management subsysten, real time indexing procedures, and a peer review setup has turn out to be a real pain in the *ss!

ukgimp




msg:485539
 8:23 am on Oct 21, 2002 (gmt 0)

What are the options for building your own directory?

Adapt and existing one. I have mentioned it before but phpHoo is pretty good. It does not come with a back end admin but the font end is pretty good

jmccormac




msg:485540
 1:02 am on Oct 22, 2002 (gmt 0)

I'd have to agree with ukgimp on this. Unless you are a programmer, it is far easier to adapt existing software. Writing your own software is not that difficult as long as you can figure out the structure of ODP. Parsing the ODP to produce static html pages is simple enough [1] but it is the edit/update back end that will cause the most problems.

Given the manner in which people now navigate a large directory site, a good search engine is an essential part of the make up of a directory as you will find that people will use the SE to navigate the site.

Regards...jmcc

[1] Parse the RDF into SQL statements for insertion into a MySQL database and then pull the data from the MySQL db to generate the static pages. It would probably work with a dynamic page generation script though the load on the server is much less with a set of static pages with a small SE.

shady




msg:485541
 1:06 am on Oct 22, 2002 (gmt 0)

Often I receive emails asking permission to add my sites the an unknown directory.

Would it be bad practice to build a directory yourself rather than waiting for submissions? After all, google adds sites without asking permission!

jmccormac




msg:485542
 2:06 am on Oct 22, 2002 (gmt 0)

In msg25 shady posted:

"Would it be bad practice to build a directory yourself rather than waiting for submissions? After all, google adds sites without asking permission!"

I don't think that many directories would exist without a lot of initial work being done by the people who create them. Relying on submissions is not really viable. Some of the directory type websites in Ireland that have relied on submissions (eg Doras) have become deadzones. The reason for this is that wonderful Catch 22 - traffic.

In order to have good submission numbers you have to have a well known site with a lot of traffic. The two common ways to do this are by providing a resource or by advertising. The advertising option tends to be expensive. By creating your own directory, you are really providing a resource for others to use. While it is often polite to ask the site owners about inclusion, most site owners would be happy at the chance of more traffic.

The problem with directories, and it is one that most face sooner rather than later, is aquisition of new sites. Google and the other search engines tend to be active (search based) rather than passive (submissions based). If your directory is going to be a niche directory then it is possible to update it manually but for a large (eg country-wide) directory, some better method of acquiring new site entries would be required. Relying on ODP is not necessarily the best thing for this as the quality from country to country varies considerably. However ODP would give your directory a quick start.

Regards...jmcc

Woz




msg:485543
 2:26 am on Oct 22, 2002 (gmt 0)

>Would it be bad practice to build a directory yourself rather than waiting for submissions?

I have many directories with many 1000s of listings. Of these less than 50 were submitted.

If you are going to build a directory then you have to build a directory. Don't wait for submissions or it will never get built.

Onya
Woz

Brad




msg:485544
 1:27 pm on Oct 22, 2002 (gmt 0)

>Would it be bad practice to build a directory yourself rather than waiting for submissions?

You have to start out with a base of sites and you will have to add those yourself. One thing that can really help is having a directory script with some sort of spider. Even if that spider only fetches meta-tags. It can save hundreds of hours of work. I run scripts both with and without a metaspider and I really appreciate having it.

Another thing you might consider is also providing a good metasearch script. This gives your searchers access to those active spidering engines right from your site. So if the directory does not provide them with what they are looking for you are still giving them a tool to find it.

Finally, think about doing what the big boys (used to) do: see about getting a backfill feed from a spidering engine just like Yahoo used to do and MSN still does.

People will come to you hoping for results and you need to meet their expectations. :)

HyperGeek




msg:485545
 2:44 pm on Oct 23, 2002 (gmt 0)

I've been developing an entertainment directory for about two years now.

Creating the base directory structure was not hard at all.

Even writing the code for the search function, advert. tracking, ect. was nothing more than time consuming.

The hardest part will be to get submissions rolling and to come up with more stuff to make our engine unique to those of our competitors. If you're thing of just throwing up a "search the directory" engine, then you're just wasting your time.

Throw some personality and user incentives in -- then maybe you may have something that has a chance of competing in today's market.

[edited by: rcjordan at 3:03 pm (utc) on Oct. 23, 2002]
[edit reason] no references to your site, please [/edit]

rcjordan




msg:485546
 5:37 pm on Oct 23, 2002 (gmt 0)

I have some experience with this. IMO, those advising caution or offering a reality check on the likely outcome being (much) less than expected are giving very sound advice.

>How viable is this profit wise? Anyone have any experience? I know that I've been working on building links to a site that would fit in a directory like this and would easily drop $20-$100 for a yearly listing if the directory was focused and well linked itself

I've been using a static directory system (Links2 with several custom modules) to categorize/manage my sponsors for 3 years or more, prior to that I used a dynamic database-driven directory. It's been my experience that small businesses really don't understand (or care) about the directory itself, they just want to buy an ad/link. The price point I chose begins at $150/yr. --less than that isn't worth the effort.

My original "vision" is still pretty much intact, but when I first started, the concept exceeded my programming capabilities or those of off-the-shelf scripts. I chose to plow ahead with the dynamic version and then address the shortfalls later. I closed most of the concept holes with the move to Links2, but had to spend many, many days hacking the code (with help, I don't code Perl) to get it where I wanted it.

My #1 mistake had little to do with programming. To lighten the administrative load I chose to bill annually. That worked fine until time for renewals, then I found that (A) no one remembered that they had subscribed and/or (B) my contact no longer worked there. We moved to monthly billing last year. Even when they ask to prepay for the entire year we send them a quarterly memo billing marked "Paid" just to keep the contact list fresh.

As for the directory itself, one of the earliest problems I had was ~you can guess this~ RANKING. While SEO (benefits of going static) was a key reason for making the conversion to Links2, the A#1 reason I changed was to be able to control the positioning within the categories. Now, rank is based on a sponsor's total monthly ad dollars spent with us. This setup also allows me to populate the categories with free, unranked listings as backfill.

My directory's Must-Have features:
* Logos in the listing
* 2-step submittal (submit, confirm --stops spam)
* Ability to alter the template for a single, specific listing
* Handle multiple related categories
* Ability to have hidden categories

This 40 message thread spans 2 pages: 40 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Directories
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved