Page is a not externally linkable
- WebmasterWorld
-- New To Web Development
---- How Do Search Engine Robots Work?

pageoneresults - 4:02 pm on Jan 3, 2007 (gmt 0)


10:06 pm on June 11, 2001
How search engines work. A primer.
[webmasterworld.com...]

Search engines consist of five discrete software components:

  1. Spider : a robotic browser like program that downloads webpages.
  2. Crawler : a wandering spider that automatically follows links found on pages.
  3. Indexer : a blender like program that dissects webpages that are downloaded by spiders.
  4. The Database : a warehouse of the pages downloaded and processed.
  5. Search Engine Results Engine : digs search results out of the database

Some questions to ponder...

  1. Do robots accept cookies?
  2. What happens if my site forces a cookie?
  3. Do robots execute JavaScript functions?
  4. Could I be doing something technically that is stopping a robot from indexing my site?
  5. How do robots interpret my page?
  6. In what order to robots index my page? What is the very first step that robot takes?

Those are some general questions that I'm sure most that are New To Web Development might have.

What questions do you have in regards to robots?

And, who has the answers to the above? ;)

As an added bonus, I finally confirmed who coined the term SERP (Search Engine Results Page). It was Brett_Tabke as confirmed in the above topic. ;)

Thread source:: http://www.webmasterworld.com/new_web_development/3206921.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com