Welcome to WebmasterWorld Guest from 54.227.110.209

Forum Moderators: goodroi

Message Too Old, No Replies

Hiding robot.txt

How to prevent robot.txt from beeing read by visitors

     

Spazio1999

10:04 am on Mar 24, 2004 (gmt 0)

10+ Year Member



I can read any robot.txt file if I type the address:

www.foo.com/robot.txt

How can I prevent robot.txt from beeing read by visitors, but only accessible by search engines?

Thank you for replying

Brett_Tabke

10:38 am on Mar 24, 2004 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



It involves setting up your webserver so that .txt files are served by a dynamic script.

deejay

10:43 am on Mar 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



*scratches head* ok.. now I'm really curious.

Why would you want to block those few humans who would think to look?

On occasion I check robots when doing link exchanges if there is no PR on the links page, just to make sure robots aren't blocked from it.

ukgimp

11:14 am on Mar 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>Why would you want to block those few humans
>>who would think to look?

You could have sensitive folders in there somewhere. While not a great level of deterent, it will at least make it one step more to find your private stuff.

Brett_Tabke

11:19 am on Mar 24, 2004 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



several huge sites have or have in the past cloaked their robots.txt (including ebay...).

CK out the msg by jd here:
[webmasterworld.com...]

Spazio1999

1:31 pm on Mar 24, 2004 (gmt 0)

10+ Year Member



>> CK out the msg by jd here:
>> [webmasterworld.com...]

Thank you, I got there useful information.

Richie0x

1:00 pm on Apr 3, 2004 (gmt 0)

10+ Year Member



I'd also like to stop normal users from accessing robots.txt.

Unfortunately my server provider doesn't allow use of htacces.

If you created a directory called 'robots.txt' and then put an 'index.php' in it, which had a script to determine whether or not the browser has 'Mozilla' or not and then echo different content depending whether it was a robot or user, would that work?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month