Forum Moderators: goodroi
directory alias, symbolics links, rewrite....will anything work?
RewriteBase /
RewriteCond %{HTTP_HOST} ^www.example.net
RewriteRule ^robots.txt$ robots_net.txt [L]RewriteCond %{HTTP_HOST} ^www.example.com
RewriteRule ^robots.txt$ robots_com.txt [L]
Should I prevent G for accessing the robots_net.txt from the .com or it doesn't matter?
I already have
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Googlebot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FAST-Crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Msnbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Msnbot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Teoma [NC]
RewriteRule .* - [F,L]
so this robots.txt was just as a backup.
what do you think?