Page is a not externally linkable
pageoneresults - 7:00 pm on Jan 26, 2002 (gmt 0)
Its a very simple step to make sure that the file validates and there is no reason to neglect having it validate 100%!
I wouldn't want to take the chance of having a robots.txt file that wasn't 100% validated... DOS Line Enders:
Another common mistake, is editing your robots.txt in DOS mode. Although it is such a common problem, that we are sure search engines account for it, it is bad practice. Always edit your robots.txt in UNIX mode and upload in ASCII. Many FTP clients will make the transformation to Unix line enders for you seamlessly, but obviously some will not. Kick your text editor into Unix mode before editing a robots.txt file.