homepage Welcome to WebmasterWorld Guest from 54.167.238.60
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots dot text for all user files?
robots.txt all public_html
kj6loh




msg:1527821
 1:28 am on Jun 11, 2004 (gmt 0)

Is there a way to include deny access to all user public_html directories, yet allow access to the main site (to spiders that is).
or do I just have to just tell the users to put a robots.txt file in their public_html directories.

 

bakedjake




msg:1527822
 3:35 pm on Jun 11, 2004 (gmt 0)

Welcome to WebmasterWorld, kj6loh. (That's a callsign I assume - KC8ORV here).

If all users are under a subdirectory, just disallow that subdirectory. For example, if you have:

/users/allan
/users/bob
/users/felix

Then just disallow /users.

kj6loh




msg:1527823
 7:19 pm on Jun 11, 2004 (gmt 0)

Ok, that's simpler than I thought then thanks. But then how will spiders/robots actually read the file? If user dirs are in /home and the www server sees /home/user/public_html, how is a robots.txt file going to be effective? Wouldn't each user have to put in in their public_html dir?
IE.
/home/usern/public_html
index.html
robots.txt
...

kj6loh




msg:1527824
 7:21 pm on Jun 11, 2004 (gmt 0)

Sorry, just a brainfart, I understand and thanks. And yes it is a callsign.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved