Welcome to WebmasterWorld Guest from 54.161.45.156
Forum Moderators: Ocean10000 & incrediBILL & phranque
I am trying to hide robots.txt from browsers but not from robots.
I am using this code in my htaccess file:
RewriteCond {HTTP_USER_AGENT} ^Mozilla
RewriteCond %{HTTP_USER_AGENT}!(Slurp¦surfsafely)
RewriteRule ^robots\.txt$ /someotherfile [L]
But then i get 501 error.
Something wrong with the code?
Thanks
Anyone who knows what a robots.txt file is and wants to read yours will be able to just spoof the user agent. I don't think that's any kind of secret hacker knowledge. And regular users will never see the robots.txt file.
If you really want to down your robots.txt file down, you may have better luck by allowing access only to IP addresses of known spiders.
That said, I'm not sure about a 501-Not Implemented error, but the missing "%" in your first RewriteCond may cause a 500-Server Error.
Jim
I derived hiding robots.txt from here:
[webmasterworld.com...]
Thanks
Just my opinion, but I think there are better things to spend your time on. Why not let your competitors see your robots.txt, but put a few extra Disallow entries in there that don't really exist, and are not linked from anywhere on the Web? Then if you ever see an access to one of those Disallowed fake URL-prefixes, you can rewrite it to a script that bans the IP address. ;)
Jim