How do you make a search engine?

the programming theory behind it

[korkus2000]

I have never really programmed any type of search, except a database search. How do you create an engine. What is the theory behind it? How do you make a site search? Is it just recursing through the files? I can't imagine that is very fast.

How would you grab remote files and search those? You build a spider that stores the pages where? What would you keep and what would you throw away? Do you keep it in a database? How do you keep the keyword density, or is that measured at search time? I have always wondered about site search and search engines programatically, but have just never asked.

Not really looking for code just the theory behind it?

[edited by: korkus2000 at 1:02 pm (utc) on Oct. 3, 2003]

[creative craig]

This is a very very basic view that Brett wrote a while back:

[webmasterworld.com...]

Craig

[ukgimp]

[webmasterworld.com...]

I got myself all over the place with this. But there are some decent links/comments in there.

[encyclo]

You could also look at the source code from Nutch - which is an open source search engine with a BSD/Apache-style license.

htt*p://www.nutch.org/