dstiles - 10:52 pm on Apr 29, 2012 (gmt 0)

I've been running a town-based directory since 1996 and a UK shopping-sites directory since 2002. I am not claiming these two are important, merely using them as a jumping-off place for the comments below.

We get a few submissions per week on each. Both are fairly small but all submissions are manually checked. Scaling this up would be reasonable only if the directories made money, which they don't. So, a way of monetising would be necessary.

In my experience it's easier to get submissions than visitors - SEOs take the trouble to find the sites and submit; users often don't know about the sites and never find them.

After accepting a site there is the problem of periodically checking to see if it is still "live". Because of parked domains which return a 200, re-used domains (one I recall used to be a pet site and is now an "adult" site) and a variety of other points of failure it is not easy to check for such things automatically. A file of (eg) parking companies (with IPs) would need to be included in the link-checker and interpretation of return codes and inherent delays would need to be taken into account. I haven't yet found a link checker that will do this so one would have to be built that checked not only the above but altered site content as well. My own solution is to periodically go through the links taking out as much as klink can find as dead and then check the rest manually, but since the directories are now almost "hobby" sites the checks do not get made that often. As I say, a good automatic site checker would need to be built and (I think) then backed up by a human.

However, the main problem is lack of people who actually use the directories. Traffic is low since few people actually type (eg) uk directory into a major search engine in the first place, so SE listings are useful only if the directory is listed for important (and probably most) keywords. Keeping people returning to the site is a serious task.

Publicity is a problem. Without money behind such a scheme publicity (eg press, TV, radio, web) is unlikely.

Someone mentioned above that only e-commerce sites need be included. I disagree: people need to find local health, community, leisure and informational sites as well. Without those any UK directory/SE would not be very useful. I can't remember the last time I ever used a comparison site other than to verify it, but I think they should also be included. And what about auction sites and similar? Ebay isn't the only auction site used in the UK.

And what about "no records found"? I think at least data should be fed from one or more engines direct through the site for items not found, world sites (which WILL be requested!) and similar: if you let them off the site they may not come back. DuckDuckGo recently became a semi-meta engine because of the problem in maintaining an index themselves (from memory - there is info on the site).

As a response to the above, I would also require, if using such a directory/SE, that "localised" searches could be easily made (eg using drop-downs), at least by town and possibly down to postcode for local-only sites (eg local shops, takeaways, restaurants etc).

Finally, there are already some UK-based engines around. These should certainly be examined.