I am looking for a search engine solution for my site. I would like to have a directory of websites where people can find sites by category, but also I would like there to be a spider that goes out and indexes the information on those sites we have put in the directory. Does anyone know of a good solution?
Perlfect Search is the best site-search solution that I know of. It has an indexer (spider) that will build an index of all the files within specified directories. The search is fast and normally fairly effective.
It won't build a directory like you mentioned, however, and for that, I suspect you'll need separate software anyway. In fact, you may just want to find some script that allows you to set up a directory; many of these should include search engines of their own (although I doubt they work by spidering content; probably the search the database).