Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt Disallow - How much time till its obeyed?

jdMorgan - 4:18 pm on Aug 6, 2003 (gmt 0)


notsleepy,

Use the WebmasterWorld server headers checker [webmasterworld.com] to determine what your Expires and Cache:max-age settings are for your existing robots.txt. Google adheres to these settings reliably (if you provide them).

Example: 

HTTP/1.1 200 OK 
Date: Wed, 06 Aug 2003 16:14:01 GMT 
Server: Rapidsite/Apa/1.3.27 (Unix) FrontPage/5.0.2.2510 mod_ssl/2.8.12 OpenSSL/0.9.7a 
Cache-Control: must-revalidate, max-age=7200 
Expires: Wed, 06 Aug 2003 18:17:30 GMT 
Last-Modified: Sat, 02 Aug 2003 05:40:41 GMT 
ETag: "1b6eae6-ac6-3f2b4ed9" 
Accept-Ranges: bytes 
Content-Length: 2758 
Connection: close 
Content-Type: text/plain

This shows that my robots.txt is to be considered valid for only two hours, and must be re-fetched if the user-agent has an older copy.

Jim

Thread source:: http://www.webmasterworld.com/robots_txt/32.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com