| How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder |
walkman
msg:1529179 | 6:48 pm on Dec 2, 2004 (gmt 0) |
is there a way to this? I know it's not convential but Yahoo has blocked one .com for something done 4-5 years ago. I have blocked the respective engines via rewrite but I also need to block Google via robots.txt from the .net just in case. directory alias, symbolics links, rewrite....will anything work?
|
dmorison
msg:1529180 | 8:19 pm on Dec 2, 2004 (gmt 0) |
rewrite should work fine in this situation; you just need two separate robots.txt format files called (for example) robots_com.txt and robots_net.txt and use rewrite something along the lines of RewriteBase / RewriteCond %{HTTP_HOST} ^www.example.net
RewriteRule ^robots.txt$ robots_net.txt [L] RewriteCond %{HTTP_HOST} ^www.example.com
RewriteRule ^robots.txt$ robots_com.txt [L] |
|
|
walkman
msg:1529181 | 9:01 pm on Dec 2, 2004 (gmt 0) |
Thank You!
worked perfectly.
I just used the first part
for the .net domain. Worked like a charm. Should I prevent G for accessing the robots_net.txt from the .com or it doesn't matter? I already have
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Googlebot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FAST-Crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Msnbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Msnbot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Teoma [NC]
RewriteRule .* - [F,L]
so this robots.txt was just as a backup. what do you think?
|
|
|
