Msg#: 3330636 posted 7:56 am on May 4, 2007 (gmt 0)
System: The following message was cut out of thread at: http://www.webmasterworld.com/robots_txt/3310622.htm [webmasterworld.com] by goodroi - 7:31 am on May 4, 2007 (utc -5) Hi, sorry for posting this noob question. Can anyone post a complete robots.txt file here? I don't know how to start and end a robots.txt file.
Msg#: 3330636 posted 12:41 pm on May 4, 2007 (gmt 0)
Robots.txt is simply a text file. There is no special beginning or ending. You just need to specify the user agent and then list what you want that user agent to disallow.
Many sites don't need a robots.txt file since the webmaster wants all content to be indexed. If you need to use a robots.txt you very likely need it customized to your personal situation. This is because your website directory/page structure is different from others.
If you want to see examples of robots.txt files just go visit your favorite websites and add robots.txt to the URL. Well that is except for [webmasterworld.com...] which has been turned into a blog.