Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
How should I use a robots.txt file on a virtual account? I want to ban spiders from one site only soccer_star msg:1528346 9:08 pm on Jun 9, 2004 (gmt 0) I have a virtual hosting account on the same IP address and have 7 sites on it. Each site has its own folder with its own index.html file in it etc. I currently have no robots.txt files at all.
I want to ban all spiders from one of my sites, but have the other 6 continued to be crawled normally. How should I go about doing this? (I am scared of accidentally banning spiders from all 7 sites).
My guess is to put the following robots.txt file in the same directory where my index.html is for the site I want ignored:
Is that correct? Or would a robots.txt file have to go in the root dir of my virtual account?
Any help would be greatly appreciated.
jdMorgan msg:1528347 9:41 pm on Jun 9, 2004 (gmt 0)
Don't think server files and directories, think URLs.
Put it wherever http:
//www.ignoredsite.com/robots.txt will access it. And yes, that's probably the same directory as the index file, but it does depend on your server configuration. But just upload it there and test by typing the robots.txt URL into your browser.
If you're worried about the other domains, then test them, too. If you haven't put a robots.txt on them, you should get a 404.
soccer_star msg:1528348 2:01 pm on Jun 10, 2004 (gmt 0)
Thanks for the tip Jim, I tested it how you suggested and the robots.txt appeared for the ignored-site.com and I got 404 for the others, so it appears to have worked. :)