Welcome to WebmasterWorld Guest from 54.162.168.187

Forum Moderators: goodroi

Message Too Old, No Replies

Hiding robot.txt

How to prevent robot.txt from beeing read by visitors

     
10:04 am on Mar 24, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 16, 2003
posts:2
votes: 0


I can read any robot.txt file if I type the address:

www.foo.com/robot.txt

How can I prevent robot.txt from beeing read by visitors, but only accessible by search engines?

Thank you for replying

10:38 am on Mar 24, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38070
votes: 16


It involves setting up your webserver so that .txt files are served by a dynamic script.
10:43 am on Mar 24, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 9, 2002
posts:861
votes: 0


*scratches head* ok.. now I'm really curious.

Why would you want to block those few humans who would think to look?

On occasion I check robots when doing link exchanges if there is no PR on the links page, just to make sure robots aren't blocked from it.

11:14 am on Mar 24, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 6, 2001
posts:2213
votes: 0


>>Why would you want to block those few humans
>>who would think to look?

You could have sensitive folders in there somewhere. While not a great level of deterent, it will at least make it one step more to find your private stuff.

11:19 am on Mar 24, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38070
votes: 16


several huge sites have or have in the past cloaked their robots.txt (including ebay...).

CK out the msg by jd here:
[webmasterworld.com...]

1:31 pm on Mar 24, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 16, 2003
posts:2
votes: 0


>> CK out the msg by jd here:
>> [webmasterworld.com...]

Thank you, I got there useful information.

1:00 pm on Apr 3, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 13, 2004
posts:144
votes: 0


I'd also like to stop normal users from accessing robots.txt.

Unfortunately my server provider doesn't allow use of htacces.

If you created a directory called 'robots.txt' and then put an 'index.php' in it, which had a script to determine whether or not the browser has 'Mozilla' or not and then echo different content depending whether it was a robot or user, would that work?

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members