g1smd - 6:01 pm on Sep 11, 2011 (gmt 0)

I found that the robots.txt in that one site had somehow become scrambled, such that all rules were on one continuous line

Are you sure they were all on one line? Perhaps the file format was using Unix-style LF line endings and you were using a text editor that required DOS/Windows-style CR/LF line endings to view it?