Welcome to WebmasterWorld Guest from 54.196.214.35

Forum Moderators: mademetop

Message Too Old, No Replies

How search engines work. A primer.

     
6:06 am on Jun 12, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


How Search Engines Work

Search engines consist of five discrete software components:

  1. spider : a robotic browser like program that downloads webpages.
  2. crawler : a wandering spider that automatically follows links found on pages.
  3. indexer : a blender like program that dissects webpages that are downloaded by spiders.
  4. the database : a warehouse of the pages downloaded and processed.
  5. search engine results engine : digs search results out of the database.

Spider:
A spider is a robotic program that downloads webpages. It works just as your browser does when you connect to a web site and download a page. The spider just doesn't have any visual components. You can see the same thing by viewing any webpage, and then selecting "view source" in your browser.

Crawler:
As a spider downloads pages, it can strip apart the page and look for "links". It is the crawlers job to then decide where the spider should go to next based on the links, or based upon a preprogrammed list of urls.

Indexer:
An indexer rips apart a page into it's various components and analyze them. Entities such as, titles, headings, links, text, constructs, bold, italic, and other style portions of a page are ripped apart and analyzed.

Database:
The database is the storage medium for all the data a search engine downloads and analyzes. This can require huge amounts of storage space.

Search Engine Results Engine:
Ah, the heart of the beast. It is the results engine's job to decide what pages matches a users search. This is the portion of a search engine you interact with when you perform a search. It is also the one part we are concerned with here.

When a user types in a keyword and does a search, the search engine decides what to match for results under varying criteria. The means with which it decides is called an algorithm. You may hear search engine optimization (SEO) professionals discuss "algos" from time to time and this is what they are referring too.

Although search engines have changed a great deal, most still match results to searches similar to the following:

  • Title: Is the keyword present in the title?
  • Domain/URL: Is the keyword present here?
  • Style: Bold, Italic, Large Headings: Is there some place on the page that the keyword is used in bold, italic, or in a Hx type heading?
  • Density: How many times does the keyword show on the page? The number of keywords in relation to page text is called Keyword Density.
  • MetaInformation: Although deprecated, some search engines till read meta keywords and meta descriptions.
  • Outbound Links: Who does the page link with and what are keywords in the link.
  • Inbound Links: Who else on the net has linked to this site? What is the words of the link? These are called "off the page" criteria because it's value is not immediately controllable by the page author.
  • Insite Links: What other pages in the users site does the page point too?

As you can see, a search engine will have to make many judgement calls based upon the entire page downloaded.

That's the abbreviated version of how a search engine works.

Ove

11:36 am on June 12, 2001 (gmt 0)

Senior Member from SE 

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 24, 2001
posts:786
votes: 0


Brett That was a good one!!!
1:18 pm on June 12, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 23, 2000
posts:1186
votes: 0


Hi Brett!

Thank you very much!
i appreciate it a lot.

For not native english speaking countries this is an excellent terms explication.

Most of the technical definitions about the language of computers, internet etc. are so new that you can't find them in a dictionary.

Angiolo

1:37 pm on June 12, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 29, 2001
posts:2945
votes: 0


Very good post Brett.......clarifies everything

At the risk of sounding completely retarded for hanging around here for so long without asking, what is a SERP?

I had to ask......

beasscr

3:42 pm on June 12, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Hey Brett...awsome job!

It's funny you had posted this information. I was thinking yesterday where I could find dictionary like type definitions about se's and seo.

Do you have anymore up your sleeve? I would love this type of format on information such as doorway pages, gateway pages, redirects, dynamic pages, subdomain pages and the sort.

If you have anymore oomph to give explanations such as the ones you have before I would greatly appreciate it. If not do you know of such a resource that may list these in dictionary format?

Crazy question but its so much easier refer to.

Thanks again for your post!

2:52 pm on June 13, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Thank you Brett,

I could not tell the difference between a spider an a crawler before. It was all robots to me.

2:57 pm on June 13, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 29, 2001
posts:2945
votes: 0


still want to know what a SERP is.....anyone want to enlighten me?
3:03 pm on June 13, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


Hi,

I think it stands for Serch Engine Result Page.

3:13 pm on June 13, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 29, 2001
posts:2945
votes: 0


thanks for clearing that up
5:33 am on June 15, 2001 (gmt 0)

Moderator from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11316
votes: 167


>>still want to know what a SERP is.....anyone want to enlighten me?<<

Brett - This term is asked about often enough that it should probably be put in the glossary... one more thing to do. I don't know how you manage it all.

I'm amazed at the quality and scope of this site, and I appreciate all the information you've shared with us, including your post above. Thanks.

6:04 am on June 15, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Thanks Robert done. When I coined Serp, I never expected that to catch on, but we needed to call them something. Added to the glossary.

You guys like stuff like the above? I didn't know if that would fly here or not. We are pretty top heavy with experienced members and I thought the above would be old hat to most. I guess it is time to get serious about a "new to se's" forum.

9:18 pm on June 22, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 27, 2001
posts:2547
votes: 0


>I guess it is time to get serious about a "new to se's" forum.
True, true.

Maybe also link to this information (your first post above) right off the home page? Anyways, this sort of thing can be helpful for those of us who get questions like this all the time ;)

1:24 am on June 23, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:June 1, 2001
posts:108
votes: 0


New to SE's forum would definitely be of great help.