Welcome to WebmasterWorld Guest from 220.127.116.11
PS- I tried "site search" to find a thread for this question here at webmasterworld but as I'm a noobie I can't figure out the google instructions posted.
if you get a 404 error...
then it's NOT in the correct place..
should be in the same directory as your index.htm (index.html,index.cfm,default.htm...etc)
First, check the spelling and case of your file name. robots.txt should be in lower case.
Also, are you sure you've placed the file in the root of your web directory?
For more info: [robotstxt.org...]
When FTp into the server this is what it looks like:
[folder logo] logs
[folder logo] mail
[folder logo] www
Hope this makes sense. too bad you can't post real pics in the posts.
There are many hosting companies and many setups. On some servers the ftp root is the same as www root. In others you have "www", "public_html", "httpdocs" and multiple other variations.
It's done that way so that you can "hide" certain files from public view. For example if in your case everything was public, surely you wouldn't want someone typing in widget.com/mail and looking throught your mail.
If you put any files below the web root, they will not be accessible via HTTP. You can put data that you want to protect. E.g. password files, mailing list databases, etc... Basically, anything that you wouldn't want to be downloaded by someone.
Web root is never the same as root of the file sistem. Web root is when you do a HTTP request to /. And the "/" is mapped by the web server to some place in the file system.
When you do the request to "/robots.txt" the web server looks in "/home/user/www/robots.txt" and serves the file to the user. As far as web server is conserned the root is "/home/user/www". It can't see below that.