jdMorgan - 1:54 pm on Sep 23, 2003 (gmt 0)

George,

You could add a RewriteRule to silently redirect to a secondary robots.txt for requests to the alternate domain:

Options +FollowSymlinks
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www\.domain\.net
RewriteCond %{HTTP_HOST} !^0\.0\.0\.1
RewriteRule ^robots\.txt$ /alternate_robots.txt [L]
#
RewriteCond %{HTTP_HOST} !^www\.domain\.net
RewriteCond %{HTTP_HOST} !^0\.0\.0\.1
RRewriteRule ^(.*)$ http://www.domain.net/$1 [R=permanent,L]

In this case, if the requested HOST is NOT your "main" domain, and the requested HOST (using IP address) is NOT your site's IP, then alternate_robots.txt will be served if robots.txt is requested, and any other requests will be redirected to your "main" domain.

You could also handle the Slurp problem specifically:

Options +FollowSymlinks
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www\.domain\.net
RewriteCond %{HTTP_HOST} !^0\.0\.0\.1
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/.*\ \(Slurp/
RewriteRule ^robots\.txt$ /alternate_robots.txt [L]
#
RewriteCond %{HTTP_HOST} !^www\.domain\.net
RewriteCond %{HTTP_HOST} !^0\.0\.0\.1
RewriteRule ^(.*)$ http://www.domain.net/$1 [R=permanent,L]

OT: You could also combine the first two RewriteCond lines in each case by using

RewriteCond %{HTTP_HOST} !^(www\.domain\.net¦^0\.0\.0\.1)

as long as you remember to edit the line to change the broken vertical pipe "¦" character to a solid vertical pipe from your keyboard.

Jim