homepage Welcome to WebmasterWorld Guest from 54.211.47.170
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
allow robots.txt access but block txt files reading
grigione




msg:4400606
 6:41 pm on Dec 22, 2011 (gmt 0)

in my htaccess file i have folllowing


<Files ~ "\.(tpl|txt)$">
Order deny,allow
Deny from all
SetEnvIfNoCase User-Agent "Googlebot" goodbot
Allow from env=goodbot
</Files>

I want block all to access tpl and txt files but permit to google bot crawler to access to robots.txt

this configuration working in a server but in an other don't working

any alternative?

 

g1smd




msg:4400625
 7:38 pm on Dec 22, 2011 (gmt 0)

Something like this to physically block access:

RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteRule \.t(xt|pl)$ - [F]


All agents should see robots.txt otherwise nothing is disallowed.

grigione




msg:4400640
 8:13 pm on Dec 22, 2011 (gmt 0)

ok thank you very much code working But only a question is possible also hide visualization of robots.txt at human eyes?

mememax




msg:4400692
 10:48 pm on Dec 22, 2011 (gmt 0)

I think the only way is to use the ip filter in htaccess but in this way you must be sure to insert all google's (and other user agents) IPs.

lucy24




msg:4400709
 12:03 am on Dec 23, 2011 (gmt 0)

Google may have already indexed your robots.txt-- this is common though not universal-- and in that case there's not much point to blocking humans. If you wanted to keep humans from knowing what's there, you could set a timer on robots.txt so the file only stays open for, say, half a second. Or a millisecond or whatever. Loads of time for a robot to assimilate it, but not enough for human eyeballs and brains.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved