Welcome to WebmasterWorld Guest from

Forum Moderators: ocean10000

Message Too Old, No Replies

An ASP Search Engine

7:54 pm on Jul 5, 2004 (gmt 0)

New User

10+ Year Member

joined:May 11, 2004
votes: 0

I would like to create add a search engine to my site that can spider external sites and index them to a database. I would like to keep this fairly small in scope, something that is highly niche in character. So far I have been trying to find something written in ASP that will store the data in Access, but I have not really found anything that is matches my requirements.

Anyway, can someone here suggest a decent search engine / bot? I have looked at FDSE, which is nice but it is not ASP. Any other scripts or products that people have tried would be much appreciated. Thanks again in advance for any help. Best Regards.

3:04 am on July 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 12, 2004
votes: 0

Check out [innerprise.net....] They have a few products that run on an MS platform. I haven't checked out their lower-end products in a while but they used to have some free products that worked great on a Windows server. Also, they have some heavy-duty products as well so you will have great options for an upgrade path.

[edited by: Brett_Tabke at 2:32 pm (utc) on July 15, 2004]
[edit reason] [webmasterworld.com...] [/edit]

2:27 pm on July 15, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Dec 6, 2002
votes: 0

ASP is not really a good language for writing spidering scripts. If anything you are looking for a vbscript I guess.

Lucene (written in Java) and [Lucene.net...] (which is Lucene ported to .NET) are good choices. HtDig is also a good choice but i don't think it runs well on windows.

Typically, search engine indexes are not stored in an RDBMS like access because it's not an efficient way to store the data. Though I believe lucene supports it through an interface.

Anyway, check out [lucene.net...] for details, or search for lucene.NET on sourceforge.

[edited by: Brett_Tabke at 2:33 pm (utc) on July 15, 2004]
[edit reason] [webmasterworld.com...] [/edit]

2:32 pm on July 15, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 62

We have so many se writers here that it is pretty hard to see a good recommendation thread on software.

Read Avi's site for the full monte:


5:18 pm on July 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 2, 2003
votes: 0

Here's a tip for you...

If you do decide to go down the asp or vb or vbscript route; watch our for IE cache overflow. I ran into this with a vb based crawler that used the Inet Objects. To get around the issue I had to put together a batch script that cleared the cache and have the crawler fire the batch on a routine basis.