Welcome to WebmasterWorld Guest from 54.226.2.31

Forum Moderators: goodroi

Message Too Old, No Replies

Help with robots.txt

     

rbecking

5:18 pm on Jan 19, 2009 (gmt 0)

5+ Year Member



Not if this can be done with robots.txt

I have two domains both pointing to the same site and files, eg www.example.com/forum.php and www.test.com/forum.php. In robots.txt I have it set which files and folders not to crawl etc and a sitemap but is it possible to stop it from crawling using the second domain and only using the first domain?

phranque

12:07 am on Jan 20, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld [webmasterworld.com], rbecking!

robots.txt doesn't know about domains and urls, only files and directories.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month