Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: goodroi
Many sites don't need a robots.txt file since the webmaster wants all content to be indexed. If you need to use a robots.txt you very likely need it customized to your personal situation. This is because your website directory/page structure is different from others.
If you want to see examples of robots.txt files just go visit your favorite websites and add robots.txt to the URL. Well that is except for [webmasterworld.com...] which has been turned into a blog.
Specifying the Sitemap location in your robots.txt file
You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line: