homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How should I use a robots.txt file on a virtual account?
I want to ban spiders from one site only

 9:08 pm on Jun 9, 2004 (gmt 0)

I have a virtual hosting account on the same IP address and have 7 sites on it. Each site has its own folder with its own index.html file in it etc. I currently have no robots.txt files at all.

I want to ban all spiders from one of my sites, but have the other 6 continued to be crawled normally. How should I go about doing this? (I am scared of accidentally banning spiders from all 7 sites).

My guess is to put the following robots.txt file in the same directory where my index.html is for the site I want ignored:

User-agent: *
Disallow: /

Is that correct? Or would a robots.txt file have to go in the root dir of my virtual account?

Any help would be greatly appreciated.



 9:41 pm on Jun 9, 2004 (gmt 0)

Don't think server files and directories, think URLs.

Put it wherever http://www.ignoredsite.com/robots.txt will access it. And yes, that's probably the same directory as the index file, but it does depend on your server configuration. But just upload it there and test by typing the robots.txt URL into your browser.

If you're worried about the other domains, then test them, too. If you haven't put a robots.txt on them, you should get a 404.



 2:01 pm on Jun 10, 2004 (gmt 0)

Thanks for the tip Jim, I tested it how you suggested and the robots.txt appeared for the ignored-site.com and I got 404 for the others, so it appears to have worked. :)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved