Welcome to WebmasterWorld Guest from 22.214.171.124
try adding robots.txt like this: domain.com/robots.txt
If you have a moderators forum for example then you wouldnt want that to be made public to the rest of the Internet. So you would ban a spider from it by using the robots.txt file.
Become a Pro Member