homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Help with robots.txt

5+ Year Member

Msg#: 3829652 posted 5:18 pm on Jan 19, 2009 (gmt 0)

Not if this can be done with robots.txt

I have two domains both pointing to the same site and files, eg www.example.com/forum.php and www.test.com/forum.php. In robots.txt I have it set which files and folders not to crawl etc and a sitemap but is it possible to stop it from crawling using the second domain and only using the first domain?



WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3829652 posted 12:07 am on Jan 20, 2009 (gmt 0)

welcome to WebmasterWorld [webmasterworld.com], rbecking!

robots.txt doesn't know about domains and urls, only files and directories.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved