Welcome to WebmasterWorld Guest from 54.197.116.116

Forum Moderators: goodroi

Message Too Old, No Replies

How should I use a robots.txt file on a virtual account?

I want to ban spiders from one site only

   
9:08 pm on Jun 9, 2004 (gmt 0)

10+ Year Member



I have a virtual hosting account on the same IP address and have 7 sites on it. Each site has its own folder with its own index.html file in it etc. I currently have no robots.txt files at all.

I want to ban all spiders from one of my sites, but have the other 6 continued to be crawled normally. How should I go about doing this? (I am scared of accidentally banning spiders from all 7 sites).

My guess is to put the following robots.txt file in the same directory where my index.html is for the site I want ignored:

User-agent: *
Disallow: /

Is that correct? Or would a robots.txt file have to go in the root dir of my virtual account?

Any help would be greatly appreciated.

9:41 pm on Jun 9, 2004 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Don't think server files and directories, think URLs.

Put it wherever http://www.ignoredsite.com/robots.txt will access it. And yes, that's probably the same directory as the index file, but it does depend on your server configuration. But just upload it there and test by typing the robots.txt URL into your browser.

If you're worried about the other domains, then test them, too. If you haven't put a robots.txt on them, you should get a 404.

Jim

2:01 pm on Jun 10, 2004 (gmt 0)

10+ Year Member



Thanks for the tip Jim, I tested it how you suggested and the robots.txt appeared for the ignored-site.com and I got 404 for the others, so it appears to have worked. :)