homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots dot text for all user files?
robots.txt all public_html

 1:28 am on Jun 11, 2004 (gmt 0)

Is there a way to include deny access to all user public_html directories, yet allow access to the main site (to spiders that is).
or do I just have to just tell the users to put a robots.txt file in their public_html directories.



 3:35 pm on Jun 11, 2004 (gmt 0)

Welcome to WebmasterWorld, kj6loh. (That's a callsign I assume - KC8ORV here).

If all users are under a subdirectory, just disallow that subdirectory. For example, if you have:


Then just disallow /users.


 7:19 pm on Jun 11, 2004 (gmt 0)

Ok, that's simpler than I thought then thanks. But then how will spiders/robots actually read the file? If user dirs are in /home and the www server sees /home/user/public_html, how is a robots.txt file going to be effective? Wouldn't each user have to put in in their public_html dir?


 7:21 pm on Jun 11, 2004 (gmt 0)

Sorry, just a brainfart, I understand and thanks. And yes it is a callsign.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved