homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder
walkman




msg:1529179
 6:48 pm on Dec 2, 2004 (gmt 0)

is there a way to this? I know it's not convential but Yahoo has blocked one .com for something done 4-5 years ago. I have blocked the respective engines via rewrite but I also need to block Google via robots.txt from the .net just in case.

directory alias, symbolics links, rewrite....will anything work?

 

dmorison




msg:1529180
 8:19 pm on Dec 2, 2004 (gmt 0)

rewrite should work fine in this situation; you just need two separate robots.txt format files called (for example) robots_com.txt and robots_net.txt and use rewrite something along the lines of

RewriteBase /

RewriteCond %{HTTP_HOST} ^www.example.net
RewriteRule ^robots.txt$ robots_net.txt [L]

RewriteCond %{HTTP_HOST} ^www.example.com
RewriteRule ^robots.txt$ robots_com.txt [L]


walkman




msg:1529181
 9:01 pm on Dec 2, 2004 (gmt 0)

Thank You!
worked perfectly.
I just used the first part
for the .net domain. Worked like a charm.

Should I prevent G for accessing the robots_net.txt from the .com or it doesn't matter?

I already have
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Googlebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Googlebot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FAST-Crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mediapartners-Google [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Msnbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Msnbot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Teoma [NC]
RewriteRule .* - [F,L]
so this robots.txt was just as a backup.

what do you think?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved