Welcome to WebmasterWorld Guest from 54.166.74.48

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

allow robots.txt access but block txt files reading

     
6:41 pm on Dec 22, 2011 (gmt 0)

New User

5+ Year Member

joined:July 15, 2010
posts: 4
votes: 0


in my htaccess file i have folllowing


<Files ~ "\.(tpl|txt)$">
Order deny,allow
Deny from all
SetEnvIfNoCase User-Agent "Googlebot" goodbot
Allow from env=goodbot
</Files>

I want block all to access tpl and txt files but permit to google bot crawler to access to robots.txt

this configuration working in a server but in an other don't working

any alternative?
7:38 pm on Dec 22, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Something like this to physically block access:

RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteRule \.t(xt|pl)$ - [F]


All agents should see robots.txt otherwise nothing is disallowed.
8:13 pm on Dec 22, 2011 (gmt 0)

New User

5+ Year Member

joined:July 15, 2010
posts: 4
votes: 0


ok thank you very much code working But only a question is possible also hide visualization of robots.txt at human eyes?
10:48 pm on Dec 22, 2011 (gmt 0)

Junior Member

5+ Year Member

joined:Aug 28, 2008
posts: 71
votes: 0


I think the only way is to use the ip filter in htaccess but in this way you must be sure to insert all google's (and other user agents) IPs.
12:03 am on Dec 23, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13531
votes: 403


Google may have already indexed your robots.txt-- this is common though not universal-- and in that case there's not much point to blocking humans. If you wanted to keep humans from knowing what's there, you could set a timer on robots.txt so the file only stays open for, say, half a second. Or a millisecond or whatever. Loads of time for a robot to assimilate it, but not enough for human eyeballs and brains.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members