Welcome to WebmasterWorld Guest from 184.72.177.182

Forum Moderators: goodroi

Message Too Old, No Replies

Help with robots.txt

     
5:18 pm on Jan 19, 2009 (gmt 0)

New User

5+ Year Member

joined:Jan 19, 2009
posts: 1
votes: 0


Not if this can be done with robots.txt

I have two domains both pointing to the same site and files, eg www.example.com/forum.php and www.test.com/forum.php. In robots.txt I have it set which files and folders not to crawl etc and a sitemap but is it possible to stop it from crawling using the second domain and only using the first domain?

12:07 am on Jan 20, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


welcome to WebmasterWorld [webmasterworld.com], rbecking!

robots.txt doesn't know about domains and urls, only files and directories.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members