pageoneresults - 4:32 pm on Jan 25, 2002 (gmt 0)

I noticed above that some said to use Note Pad to set up the robots.txt file or FP. Unfortunately you cannot do this and have a file that validates 100%. You will end up with a warning that reads...

Robots.txt contains DOS or Mac line enders. Although it is so common most spiders will deal correctly with the file, it is not a valid format.

You can use Note Pad to set it up, but you will need to upload via FTP using the ASCII mode so that the above DOS or Mac line enders are not present. I learned this after numerous testing runs and using Brett's robots.txt validator.

You can also use a number of other programs that support UNIX mode.

I'm not sure if having a robots.txt file in place gives you any advantage other than blocking those spiders that obey the directives. I keep one present in all web site directories that I manage just to minimize the 404 errors and to keep them out of certain sub directories like css and javascript.

I've read many topics on this during the past couple of years and I've seen too many comments about a spider calling the robots.txt file, not finding it, and then moving on its merry way never to return. Its a simple file to set up and I think all web sites should have one present in their root directory, even if its just the standard...

User-agent: *
Disallow:

(edited by: pageoneresults at 4:36 pm (utc) on Jan. 25, 2002)