homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

allow robots.txt access but block txt files reading

 6:41 pm on Dec 22, 2011 (gmt 0)

in my htaccess file i have folllowing

<Files ~ "\.(tpl|txt)$">
Order deny,allow
Deny from all
SetEnvIfNoCase User-Agent "Googlebot" goodbot
Allow from env=goodbot

I want block all to access tpl and txt files but permit to google bot crawler to access to robots.txt

this configuration working in a server but in an other don't working

any alternative?



 7:38 pm on Dec 22, 2011 (gmt 0)

Something like this to physically block access:

RewriteCond %{REQUEST_URI} !^/robots.txt$
RewriteRule \.t(xt|pl)$ - [F]

All agents should see robots.txt otherwise nothing is disallowed.


 8:13 pm on Dec 22, 2011 (gmt 0)

ok thank you very much code working But only a question is possible also hide visualization of robots.txt at human eyes?


 10:48 pm on Dec 22, 2011 (gmt 0)

I think the only way is to use the ip filter in htaccess but in this way you must be sure to insert all google's (and other user agents) IPs.


 12:03 am on Dec 23, 2011 (gmt 0)

Google may have already indexed your robots.txt-- this is common though not universal-- and in that case there's not much point to blocking humans. If you wanted to keep humans from knowing what's there, you could set a timer on robots.txt so the file only stays open for, say, half a second. Or a millisecond or whatever. Loads of time for a robot to assimilate it, but not enough for human eyeballs and brains.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved