Welcome to WebmasterWorld Guest from 54.205.96.97

Forum Moderators: Ocean10000 & incrediBILL & phranque

allow robots.txt access but block txt files reading

   
6:41 pm on Dec 22, 2011 (gmt 0)



in my htaccess file i have folllowing


<Files ~ "\.(tpl|txt)$">
Order deny,allow
Deny from all
SetEnvIfNoCase User-Agent "Googlebot" goodbot
Allow from env=goodbot
</Files>

I want block all to access tpl and txt files but permit to google bot crawler to access to robots.txt

this configuration working in a server but in an other don't working

any alternative?
7:38 pm on Dec 22, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Something like this to physically block access:

RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteRule \.t(xt|pl)$ - [F]


All agents should see robots.txt otherwise nothing is disallowed.
8:13 pm on Dec 22, 2011 (gmt 0)



ok thank you very much code working But only a question is possible also hide visualization of robots.txt at human eyes?
10:48 pm on Dec 22, 2011 (gmt 0)

5+ Year Member



I think the only way is to use the ip filter in htaccess but in this way you must be sure to insert all google's (and other user agents) IPs.
12:03 am on Dec 23, 2011 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



Google may have already indexed your robots.txt-- this is common though not universal-- and in that case there's not much point to blocking humans. If you wanted to keep humans from knowing what's there, you could set a timer on robots.txt so the file only stays open for, say, half a second. Or a millisecond or whatever. Loads of time for a robot to assimilate it, but not enough for human eyeballs and brains.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month