Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Hiding robots.txt


jdMorgan - 5:45 am on Oct 29, 2003 (gmt 0)


keyplyr,

> I understand the last line, but would you explain the first two?

RewriteCond %{HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT} !(Slurp¦surfsafely)
RewriteRule ^robots\.txt$ /someotherfile [L]

Line 1: IF the User-agent string starts with "Mozilla" (most browsers)
Line 2: AND IF the User-agent string does not contain "Slurp" or "surfsafely" (Two 'bots w/UAs that start with "Mozilla")
Line 3: THEN do the rewrite of robots.txt to some other file.

(Note correction of code line 1 from first post. Also, replace "¦" with a solid vertical pipe character from your keyboard.)

Jim


Thread source:: http://www.webmasterworld.com/robots_txt/74.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com