Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

How do I create two robots.txt for the same directory?

.com and .net are for two different engines but point at the same folder



6:48 pm on Dec 2, 2004 (gmt 0)

is there a way to this? I know it's not convential but Yahoo has blocked one .com for something done 4-5 years ago. I have blocked the respective engines via rewrite but I also need to block Google via robots.txt from the .net just in case.

directory alias, symbolics links, rewrite....will anything work?


8:19 pm on Dec 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

rewrite should work fine in this situation; you just need two separate robots.txt format files called (for example) robots_com.txt and robots_net.txt and use rewrite something along the lines of

RewriteBase /

RewriteCond %{HTTP_HOST} ^www.example.net
RewriteRule ^robots.txt$ robots_net.txt [L]

RewriteCond %{HTTP_HOST} ^www.example.com
RewriteRule ^robots.txt$ robots_com.txt [L]


9:01 pm on Dec 2, 2004 (gmt 0)

Thank You!
worked perfectly.
I just used the first part
for the .net domain. Worked like a charm.

Should I prevent G for accessing the robots_net.txt from the .com or it doesn't matter?

I already have
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Googlebot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FAST-Crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Msnbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Msnbot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Teoma [NC]
RewriteRule .* - [F,L]
so this robots.txt was just as a backup.

what do you think?


Featured Threads

Hot Threads This Week

Hot Threads This Month