ohios - 6:54 pm on Jan 26, 2002 (gmt 0)

Some points about robots.txt, with reference to [robotstxt.org...]

1. "[an empty robots.txt] ...will be treated as if it was not present, i.e. all robots will consider themselves welcome"

If robots.txt is not present then its pretty much up to the robot to make its own decision on its next action - supply a robots.txt if you think that a search engine is by passing your site due to a 404 for /robots.txt, but don't assume a missing robots.txt will stop robots coming in,

2. If the robot does find /robots.txt then its important that it is correctly formatted. Again, its up to the robot to take a decision on what to do next if the robots.txt file contains a syntax error. The author of the robot might give the benefit of the doubt and try to work out the meaning, but more than likely will ignore an incorrect robots.txt and carry on as if it wasn't there. [searchengineworld.com...] shows some common syntax errors,

3. Some syntax points from the earlier posts:

“The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL).”

CR (Mac), CRLF(Windows/MS-DOS) and LF(Unix) are all valid line terminators, so notepad and FP 2002 are valid editors for creating robots.txt with respect to end of line characters.

“The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome”

You are allowed a robots.txt that has no contents.

So, as for the original questions, I would agree with jlara that a blank robots.txt is the thing to do if you are worried that robots might be ignoring you, and you should have no problem creating robots.txt in FP 2002.