homepage Welcome to WebmasterWorld Guest from 54.198.9.77
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Help with robots.txt
rbecking




msg:3829654
 5:18 pm on Jan 19, 2009 (gmt 0)

Not if this can be done with robots.txt

I have two domains both pointing to the same site and files, eg www.example.com/forum.php and www.test.com/forum.php. In robots.txt I have it set which files and folders not to crawl etc and a sitemap but is it possible to stop it from crawling using the second domain and only using the first domain?

 

phranque




msg:3829956
 12:07 am on Jan 20, 2009 (gmt 0)

welcome to WebmasterWorld [webmasterworld.com], rbecking!

robots.txt doesn't know about domains and urls, only files and directories.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved