homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Hiding robot.txt
How to prevent robot.txt from beeing read by visitors
Spazio1999




msg:1529359
 10:04 am on Mar 24, 2004 (gmt 0)

I can read any robot.txt file if I type the address:

www.foo.com/robot.txt

How can I prevent robot.txt from beeing read by visitors, but only accessible by search engines?

Thank you for replying

 

Brett_Tabke




msg:1529360
 10:38 am on Mar 24, 2004 (gmt 0)

It involves setting up your webserver so that .txt files are served by a dynamic script.

deejay




msg:1529361
 10:43 am on Mar 24, 2004 (gmt 0)

*scratches head* ok.. now I'm really curious.

Why would you want to block those few humans who would think to look?

On occasion I check robots when doing link exchanges if there is no PR on the links page, just to make sure robots aren't blocked from it.

ukgimp




msg:1529362
 11:14 am on Mar 24, 2004 (gmt 0)

>>Why would you want to block those few humans
>>who would think to look?

You could have sensitive folders in there somewhere. While not a great level of deterent, it will at least make it one step more to find your private stuff.

Brett_Tabke




msg:1529363
 11:19 am on Mar 24, 2004 (gmt 0)

several huge sites have or have in the past cloaked their robots.txt (including ebay...).

CK out the msg by jd here:
[webmasterworld.com...]

Spazio1999




msg:1529364
 1:31 pm on Mar 24, 2004 (gmt 0)

>> CK out the msg by jd here:
>> [webmasterworld.com...]

Thank you, I got there useful information.

Richie0x




msg:1529365
 1:00 pm on Apr 3, 2004 (gmt 0)

I'd also like to stop normal users from accessing robots.txt.

Unfortunately my server provider doesn't allow use of htacces.

If you created a directory called 'robots.txt' and then put an 'index.php' in it, which had a script to determine whether or not the browser has 'Mozilla' or not and then echo different content depending whether it was a robot or user, would that work?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved