I would like to create add a search engine to my site that can spider external sites and index them to a database. I would like to keep this fairly small in scope, something that is highly niche in character. So far I have been trying to find something written in ASP that will store the data in Access, but I have not really found anything that is matches my requirements.
Anyway, can someone here suggest a decent search engine / bot? I have looked at FDSE, which is nice but it is not ASP. Any other scripts or products that people have tried would be much appreciated. Thanks again in advance for any help. Best Regards.
Check out [innerprise.net....] They have a few products that run on an MS platform. I haven't checked out their lower-end products in a while but they used to have some free products that worked great on a Windows server. Also, they have some heavy-duty products as well so you will have great options for an upgrade path.
[edited by: Brett_Tabke at 2:32 pm (utc) on July 15, 2004] [edit reason] [webmasterworld.com...] [/edit]
If you do decide to go down the asp or vb or vbscript route; watch our for IE cache overflow. I ran into this with a vb based crawler that used the Inet Objects. To get around the issue I had to put together a batch script that cleared the cache and have the crawler fire the batch on a routine basis.