Page is a not externally linkable
- WebmasterWorld
-- New To Web Development
---- How Do Search Engine Robots Work?

coopster - 12:06 am on Jan 9, 2007 (gmt 0)


Once the multi-tasking bot requests a URI, the first thing it does is request the server headers, correct?

I don't know if it still holds true, but I believe the steps are actually ...

  1. Look up DNS (possibly from local cache)
  2. Connect to host
  3. Send request
  4. Receive Response

As mentioned, it may only be a HEAD method that the robot requested. Therefore the only information received would be the headers. Otherwise yes, the header information is always passed first followed by the message-body.

So, the robot either requests just the headers, or it may request the headers and message-body.

Thread source:: http://www.webmasterworld.com/new_web_development/3206921.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com