| Help with robots.txt
|
rbecking
msg:3829654 | 5:18 pm on Jan 19, 2009 (gmt 0) |
Not if this can be done with robots.txt I have two domains both pointing to the same site and files, eg www.example.com/forum.php and www.test.com/forum.php. In robots.txt I have it set which files and folders not to crawl etc and a sitemap but is it possible to stop it from crawling using the second domain and only using the first domain?
|
phranque
msg:3829956 | 12:07 am on Jan 20, 2009 (gmt 0) |
welcome to WebmasterWorld [webmasterworld.com], rbecking! robots.txt doesn't know about domains and urls, only files and directories.
|
|
|
