homepage Welcome to WebmasterWorld Guest from 54.163.70.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
DOS Line Enders
How to download sample robots.txt files
webman2




msg:1525819
 12:55 pm on Feb 13, 2004 (gmt 0)

There is a comment in "Our big crawl" about how to correctly edit (in UNIX mode) and upload (in ASCII) robots.txt files.

This becomes relevant with the sample robots4.txt as the lines are long. I don't know how to put my editor into UNIX mode and I have no need to modify this file as it was created by experts. What is the correct way to download and save it please, so that it is not corrupted? I use Windows OS.

Thanks

 

DaveAtIFG




msg:1525820
 5:34 pm on Feb 13, 2004 (gmt 0)

I think msg #2 in this thread [webmasterworld.com] will clarify for you. At least I hope so! ;)

webman2




msg:1525821
 10:30 am on Feb 14, 2004 (gmt 0)

Thanks Dave,

But if FTP transfers corrects the mistakes, I was wondering why robots4.txt posted on this site doesn't validate with the validator on this site.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved