Can anyone post a complete robots.txt file here?

[PowerUp]

---

System: The following message was cut out of thread at: http://www.webmasterworld.com/robots_txt/3310622.htm [webmasterworld.com] by goodroi - 7:31 am on May 4, 2007 (utc -5)

---

Hi, sorry for posting this noob question. Can anyone post a complete robots.txt file here? I don't know how to start and end a robots.txt file.

[goodroi]

Robots.txt is simply a text file. There is no special beginning or ending. You just need to specify the user agent and then list what you want that user agent to disallow.

Many sites don't need a robots.txt file since the webmaster wants all content to be indexed. If you need to use a robots.txt you very likely need it customized to your personal situation. This is because your website directory/page structure is different from others.

If you want to see examples of robots.txt files just go visit your favorite websites and add robots.txt to the URL. Well that is except for [webmasterworld.com...] which has been turned into a blog.

[PowerUp]

Then does that mean if I want the robot to follow my sitemap, all I need to do is add the following line and nothing else... and save it as txt?

"Sitemap: http://www.example.com/sitemap.xml"

[goodroi]

As per [sitemaps.org...]

Specifying the Sitemap location in your robots.txt file
You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line:
Sitemap: <sitemap_location>