Forum Moderators: open
Thanks in advance,
Allan
[edited by: NFFC at 1:55 pm (utc) on Sep. 22, 2002]
[edit reason] Live URL's removed [/edit]
P.S. A blank robots.txt file is not recommended. According to Brett, some robots may misinterpret a blank robots text file.
Having no robots.txt is ok though, right, although probably not preferrable. I'm assuming 404 error is presented properly by the server.
It is, although I've not worked without one for at least 3 years now. Not having one is the same as having an empty one (with the exception of the misinterpretation that Brett mentioned).
Having one also eliminated all the 404 errors we were getting for the requested robots.txt file.
User-agent: *
Disallow: /_private/
Disallow: /css/
Disallow: /javascript/
P.S. I think the best bet for you right now is to remove the robots.txt altogether until you have a better understanding of how it works and what it is used for. ;)
User-agent: *
The biggest advantage of the second file mentoned over not having a robots.txt file at all is that it will save you bandwidth. We all hear horror stories about mysterious bots grabbing entire sites and downloading 100000000000 files an hour... If you stick to the known bots you will usualy avoid this.
Something was there, I checked the robots.txt and got an empty file. It did not return a 404. Just went and checked again, this is what is in your robots.txt file...
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=windows-1252"></HEAD>
<BODY></BODY></HTML>
You cannot have that in the file, it is not an html file. Again, I would do a little studying on the use of the robots.txt and how to properly format it before uploading one to the root of your directory.
P.S. I have no clue how a robot would react to the way your file is set up. Maybe Brett can shed some light.
