Welcome to WebmasterWorld Guest from 54.196.147.57

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

allow robots.txt access but block txt files reading

     

grigione

6:41 pm on Dec 22, 2011 (gmt 0)

5+ Year Member



in my htaccess file i have folllowing


<Files ~ "\.(tpl|txt)$">
Order deny,allow
Deny from all
SetEnvIfNoCase User-Agent "Googlebot" goodbot
Allow from env=goodbot
</Files>

I want block all to access tpl and txt files but permit to google bot crawler to access to robots.txt

this configuration working in a server but in an other don't working

any alternative?

g1smd

7:38 pm on Dec 22, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Something like this to physically block access:

RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteRule \.t(xt|pl)$ - [F]


All agents should see robots.txt otherwise nothing is disallowed.

grigione

8:13 pm on Dec 22, 2011 (gmt 0)

5+ Year Member



ok thank you very much code working But only a question is possible also hide visualization of robots.txt at human eyes?

mememax

10:48 pm on Dec 22, 2011 (gmt 0)

5+ Year Member



I think the only way is to use the ip filter in htaccess but in this way you must be sure to insert all google's (and other user agents) IPs.

lucy24

12:03 am on Dec 23, 2011 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



Google may have already indexed your robots.txt-- this is common though not universal-- and in that case there's not much point to blocking humans. If you wanted to keep humans from knowing what's there, you could set a timer on robots.txt so the file only stays open for, say, half a second. Or a millisecond or whatever. Loads of time for a robot to assimilate it, but not enough for human eyeballs and brains.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month