The cost to install and configure a script on your server for this is quite likely to be greater than the cost to use one of the lower-cost affiliate networks; I like ShareASale (and used it to launch a new affiliate program last fall). Their cost is relatively low $250 setup, $100 minimum deposit against commissions and fees; and their fee is 20% of the amount paid to affiliates.
If you're looking for a solution that can also be used for your own PPC and paid advertising, then a locally-hosted script may make sense, but unfortunately the cost of a quality script with good support is going to be much greater.
I am in a similar situation as danaj. We have a niche product and receive interest in an affiliate program, too.
I think it would be helpful and interesting to hear the opinions of those who participate actively in affiliate programs: - Does using a 3rd party site (like shareasale.com) over a script hosted by the publisher increase credibility and chances of an affiliate participating? - Will using a 3rd party site increase the number of affiliates who sign up (and sell) because they find you listed in the directory? - Does accepting phone orders significantly decrease affiliate participation?