Welcome to WebmasterWorld Guest from 54.205.168.74

Forum Moderators: goodroi

Message Too Old, No Replies

Robots dot text for all user files?

robots.txt all public_html

     
1:28 am on Jun 11, 2004 (gmt 0)

New User

10+ Year Member

joined:June 11, 2004
posts:7
votes: 0


Is there a way to include deny access to all user public_html directories, yet allow access to the main site (to spiders that is).
or do I just have to just tell the users to put a robots.txt file in their public_html directories.
3:35 pm on June 11, 2004 (gmt 0)

Administrator from CA 

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 8, 2003
posts:3878
votes: 57


Welcome to WebmasterWorld, kj6loh. (That's a callsign I assume - KC8ORV here).

If all users are under a subdirectory, just disallow that subdirectory. For example, if you have:

/users/allan
/users/bob
/users/felix

Then just disallow /users.

7:19 pm on June 11, 2004 (gmt 0)

New User

10+ Year Member

joined:June 11, 2004
posts:7
votes: 0


Ok, that's simpler than I thought then thanks. But then how will spiders/robots actually read the file? If user dirs are in /home and the www server sees /home/user/public_html, how is a robots.txt file going to be effective? Wouldn't each user have to put in in their public_html dir?
IE.
/home/usern/public_html
index.html
robots.txt
...
7:21 pm on June 11, 2004 (gmt 0)

New User

10+ Year Member

joined:June 11, 2004
posts:7
votes: 0


Sorry, just a brainfart, I understand and thanks. And yes it is a callsign.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members