homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
Forum Library, Charter, Moderators: mademetop

General Search Engine Marketing Issues Forum

How search engines work. A primer.

 6:06 am on Jun 12, 2001 (gmt 0)

How Search Engines Work

Search engines consist of five discrete software components:

  1. spider : a robotic browser like program that downloads webpages.
  2. crawler : a wandering spider that automatically follows links found on pages.
  3. indexer : a blender like program that dissects webpages that are downloaded by spiders.
  4. the database : a warehouse of the pages downloaded and processed.
  5. search engine results engine : digs search results out of the database.

A spider is a robotic program that downloads webpages. It works just as your browser does when you connect to a web site and download a page. The spider just doesn't have any visual components. You can see the same thing by viewing any webpage, and then selecting "view source" in your browser.

As a spider downloads pages, it can strip apart the page and look for "links". It is the crawlers job to then decide where the spider should go to next based on the links, or based upon a preprogrammed list of urls.

An indexer rips apart a page into it's various components and analyze them. Entities such as, titles, headings, links, text, constructs, bold, italic, and other style portions of a page are ripped apart and analyzed.

The database is the storage medium for all the data a search engine downloads and analyzes. This can require huge amounts of storage space.

Search Engine Results Engine:
Ah, the heart of the beast. It is the results engine's job to decide what pages matches a users search. This is the portion of a search engine you interact with when you perform a search. It is also the one part we are concerned with here.

When a user types in a keyword and does a search, the search engine decides what to match for results under varying criteria. The means with which it decides is called an algorithm. You may hear search engine optimization (SEO) professionals discuss "algos" from time to time and this is what they are referring too.

Although search engines have changed a great deal, most still match results to searches similar to the following:

  • Title: Is the keyword present in the title?
  • Domain/URL: Is the keyword present here?
  • Style: Bold, Italic, Large Headings: Is there some place on the page that the keyword is used in bold, italic, or in a Hx type heading?
  • Density: How many times does the keyword show on the page? The number of keywords in relation to page text is called Keyword Density.
  • MetaInformation: Although deprecated, some search engines till read meta keywords and meta descriptions.
  • Outbound Links: Who does the page link with and what are keywords in the link.
  • Inbound Links: Who else on the net has linked to this site? What is the words of the link? These are called "off the page" criteria because it's value is not immediately controllable by the page author.
  • Insite Links: What other pages in the users site does the page point too?

As you can see, a search engine will have to make many judgement calls based upon the entire page downloaded.

That's the abbreviated version of how a search engine works.



 11:36 am on Jun 12, 2001 (gmt 0)

Brett That was a good one!!!


 1:18 pm on Jun 12, 2001 (gmt 0)

Hi Brett!

Thank you very much!
i appreciate it a lot.

For not native english speaking countries this is an excellent terms explication.

Most of the technical definitions about the language of computers, internet etc. are so new that you can't find them in a dictionary.



 1:37 pm on Jun 12, 2001 (gmt 0)

Very good post Brett.......clarifies everything

At the risk of sounding completely retarded for hanging around here for so long without asking, what is a SERP?

I had to ask......


 3:42 pm on Jun 12, 2001 (gmt 0)

Hey Brett...awsome job!

It's funny you had posted this information. I was thinking yesterday where I could find dictionary like type definitions about se's and seo.

Do you have anymore up your sleeve? I would love this type of format on information such as doorway pages, gateway pages, redirects, dynamic pages, subdomain pages and the sort.

If you have anymore oomph to give explanations such as the ones you have before I would greatly appreciate it. If not do you know of such a resource that may list these in dictionary format?

Crazy question but its so much easier refer to.

Thanks again for your post!


 2:52 pm on Jun 13, 2001 (gmt 0)

Thank you Brett,

I could not tell the difference between a spider an a crawler before. It was all robots to me.


 2:57 pm on Jun 13, 2001 (gmt 0)

still want to know what a SERP is.....anyone want to enlighten me?


 3:03 pm on Jun 13, 2001 (gmt 0)


I think it stands for Serch Engine Result Page.


 3:13 pm on Jun 13, 2001 (gmt 0)

thanks for clearing that up

Robert Charlton

 5:33 am on Jun 15, 2001 (gmt 0)

>>still want to know what a SERP is.....anyone want to enlighten me?<<

Brett - This term is asked about often enough that it should probably be put in the glossary... one more thing to do. I don't know how you manage it all.

I'm amazed at the quality and scope of this site, and I appreciate all the information you've shared with us, including your post above. Thanks.


 6:04 am on Jun 15, 2001 (gmt 0)

Thanks Robert done. When I coined Serp, I never expected that to catch on, but we needed to call them something. Added to the glossary.

You guys like stuff like the above? I didn't know if that would fly here or not. We are pretty top heavy with experienced members and I thought the above would be old hat to most. I guess it is time to get serious about a "new to se's" forum.


 9:18 pm on Jun 22, 2001 (gmt 0)

>I guess it is time to get serious about a "new to se's" forum.
True, true.

Maybe also link to this information (your first post above) right off the home page? Anyways, this sort of thing can be helpful for those of us who get questions like this all the time ;)


 1:24 am on Jun 23, 2001 (gmt 0)

New to SE's forum would definitely be of great help.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved