Forum Moderators: goodroi
I just put a .htaccess file in the root of my file manager - meaning it's not in any other folder/directory, including public_html. It's all stuff like
"RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]"
etc.
What I'd like to know is: Can I add stuff from a suggested robots.txt file, like
"Disallow: /mywebsite/cgi-bin/
Disallow: /mywebsite/style_images/", etc.
to my current .htaccess? Or do I need a separate robots.txt file?
Either way, both should be in the root - meaning outside of public_html, right?
I read [webmasterworld.com...] ... .
.htaccess file and robotx should be kept separate
.htaccess file is for Apache Server
.htaccess file can be kept in any folder inside public_html or in that folder itself if you want to apply some rules to all folders inside public_html
if you want to apply some conditions in some specific folder only you can just move the htaccess file in that folder too
Robots.txt file tells the search engine to if you want to allow the website pages to be read by them? or are search engines allowed to ccess specific part of website
So you can see that both of the files have some specific functions and should be kept different