Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Help with robots.txt



5:18 pm on Jan 19, 2009 (gmt 0)

5+ Year Member

Not if this can be done with robots.txt

I have two domains both pointing to the same site and files, eg www.example.com/forum.php and www.test.com/forum.php. In robots.txt I have it set which files and folders not to crawl etc and a sitemap but is it possible to stop it from crawling using the second domain and only using the first domain?


12:07 am on Jan 20, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

welcome to WebmasterWorld [webmasterworld.com], rbecking!

robots.txt doesn't know about domains and urls, only files and directories.


Featured Threads

Hot Threads This Week

Hot Threads This Month