An ASP Search Engine

Forum Moderators: open

Message Too Old, No Replies

An ASP Search Engine

trigate

7:54 pm on Jul 5, 2004 (gmt 0)

I would like to create add a search engine to my site that can spider external sites and index them to a database. I would like to keep this fairly small in scope, something that is highly niche in character. So far I have been trying to find something written in ASP that will store the data in Access, but I have not really found anything that is matches my requirements.

Anyway, can someone here suggest a decent search engine / bot? I have looked at FDSE, which is nice but it is not ASP. Any other scripts or products that people have tried would be much appreciated. Thanks again in advance for any help. Best Regards.

dataguy

3:04 am on Jul 15, 2004 (gmt 0)

Check out [innerprise.net....] They have a few products that run on an MS platform. I haven't checked out their lower-end products in a while but they used to have some free products that worked great on a Windows server. Also, they have some heavy-duty products as well so you will have great options for an upgrade path.

[edited by: Brett_Tabke at 2:32 pm (utc) on July 15, 2004]
[edit reason] [webmasterworld.com...] [/edit]

webdevsf

2:27 pm on Jul 15, 2004 (gmt 0)

ASP is not really a good language for writing spidering scripts. If anything you are looking for a vbscript I guess.

Lucene (written in Java) and [Lucene.net...] (which is Lucene ported to .NET) are good choices. HtDig is also a good choice but i don't think it runs well on windows.

Typically, search engine indexes are not stored in an RDBMS like access because it's not an efficient way to store the data. Though I believe lucene supports it through an interface.

Anyway, check out [lucene.net...] for details, or search for lucene.NET on sourceforge.

[edited by: Brett_Tabke at 2:33 pm (utc) on July 15, 2004]
[edit reason] [webmasterworld.com...] [/edit]

Brett_Tabke

2:32 pm on Jul 15, 2004 (gmt 0)

We have so many se writers here that it is pretty hard to see a good recommendation thread on software.

Read Avi's site for the full monte:

[searchtools.com...]

Easy_Coder

5:18 pm on Jul 15, 2004 (gmt 0)

Here's a tip for you...

If you do decide to go down the asp or vb or vbscript route; watch our for IE cache overflow. I ran into this with a vb based crawler that used the Inet Objects. To get around the issue I had to put together a batch script that cleared the cache and have the crawler fire the batch on a routine basis.