Forum Moderators: bakedjake

Message Too Old, No Replies

New Search Engine

Little old retired web slave needs advice

         

Dumpy

3:28 pm on May 31, 2002 (gmt 0)

10+ Year Member



I have been working for several years on developing a search engine...well to be honest I have been working on adapting a "free" search engine to the net.

I have used the dmoz rdf dump, parsing it and searching the parsed results...using my own criteria. The volume of use has risen over time to be quite profitable.

I now am considering expanding to spidering the sites within the rdf dump...and adding a submit site page...however that worries me.

I have so far operated on the theory that all operations must be software controlled (this makes it possible for one little old fat retired guy to do everything). My software for adding sites gives me many controls on what can be added, but it still does not have eyeballs.

Does anyone feel that it is possible to place adequate software checks to prevent gross spamming?

Should require a link or button?

My system works so smooth now, I almost hate to expand.

BUT, I am empire oriented.

Brad

4:44 pm on May 31, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Should require a link or button?

I can't help you with the software end of it since I do most things manually. But I do know a bit about this. I send out a confirmation email to everyone accepted. I ask, politely, for a link back and give them a URL to my "linktous" page. But I never require a link. Over time you will start getting a certain percentage of people either linking back or adding your searchbox.

Some people demand link-backs and I think this build resentment plus creates a review nightmare.

Also: if they feel they really got a lot of value out of submitting their URL to you they are more likely to submit. Give them some sort of free goodies as a thank you. The more jaded pro web designers are unlikely to be moved by what you do unless you send them tons of traffic, but the amateurs can be increadable loyal customers if you cater to them and help them out. Long term they can send you traffic and exposure.

Last: copy the smaller directories and give them an *incentive* to put a button on their site. Like a vote button or rate this site.

Right now I give out a rate this site code that will very slowly improve a site's standing in the SERP's. It's a way for a hoo-hum site to improve it's placement and gain more exposure.

rcjordan

4:59 pm on May 31, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>BUT, I am empire oriented.

You and I are on the same wavelength, Dumpy.... nothing better than an empire.

Yes, I'd allow additions but you are wise to be on guard for spam from the git-go. If I had the programming expertise, I'd add the ransom-note password, a 2nd submittal button on a proofread/confirmation page, and require the form to be submitted from your domain. I'd also disable html in the body of the post and scan for objectionable keywords.

Brad's point about rewards is certainly valid. But, if you want to reward the savvy, just give them a good, solid, spiderable link.

<added>
On my directory, I also only allow the root index to be submitted. Deep-linking is allowed only with my approval. No free sites, geocities, tripod, etc.

Brad

12:29 am on Jun 1, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>On my directory, I also only allow the root index to be submitted. Deep-linking is allowed only with my approval. No free sites, geocities, tripod, etc.

This is tricky. You need to look at what market is there and what you want to go after.

For some directories listing free hosted sites is their very purpose - to get traffic to those sites that are not likely to know much about search engine placement. But as rcjordan indicates, there is a lot of churn in the free hosted sites so they do create some extra work.

Allowing deep linking is also tricky and depends upon the accepted practices for your market. (I like the ODP philosophy on that.)

<aside>
Of course if enough SE's/Directories refuse to list free sites or third level domains that can create a new market for a SE that will.
</aside>

WindSun

2:14 am on Jun 1, 2002 (gmt 0)

10+ Year Member



BTW, if you are using the ODP RDF dump without giving them credit, you may have another problem.

mack

4:04 am on Jun 1, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



On my directory I allow free listings to any site with family friendly content. Only to the homepage though, I dont link directly to internal pages. I send out an email notifying them that their site has been included. I explain that they are not required to lnk back but return links are always apreciated. i also send offer a link to a page with my "link to me" code. I find that about 1-50 websites will give me a return link. I spend quite a lot of time browsing and submitting to directories. If a site was to insist i return a link to them, i would ignore it.

Dumpy

9:28 pm on Jul 6, 2002 (gmt 0)

10+ Year Member



I added a "add your site" page to my search engine, with instant addition to the directory. But, after the monthly "batch" update...I have decided I will wait until the software for "on the fly" updating is completed and solid, before I venture out onto the web. I will need another server with a bunch of 72 gig drives.

I used the idea of only one site per domain...if you add another site it just wipes out the one before it. I appreciate the suggestions.