Welcome to WebmasterWorld Guest from 107.20.54.98

Forum Moderators: goodroi

Message Too Old, No Replies

DOS Line Enders

How to download sample robots.txt files

     

webman2

12:55 pm on Feb 13, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


There is a comment in "Our big crawl" about how to correctly edit (in UNIX mode) and upload (in ASCII) robots.txt files.

This becomes relevant with the sample robots4.txt as the lines are long. I don't know how to put my editor into UNIX mode and I have no need to modify this file as it was created by experts. What is the correct way to download and save it please, so that it is not corrupted? I use Windows OS.

Thanks

5:34 pm on Feb 13, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


I think msg #2 in this thread [webmasterworld.com] will clarify for you. At least I hope so! ;)

webman2

10:30 am on Feb 14, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


Thanks Dave,

But if FTP transfers corrects the mistakes, I was wondering why robots4.txt posted on this site doesn't validate with the validator on this site.