Robots.txt

[allanforbes]

I want to add a robots.txt file to my site- should it appear on the site map?

[Dreamquick]

Not really - the only things that need to read robots.txt are crawlers and they already know where it lives (/robots.txt) so there's no point referencing it on the site itself.

- Tony

[allanforbes]

Thanks Tony

[GoldDust]

I have chosen to leave the robot txt and go for the meta tag instead!

Is this a good chioce when i dont have any pages to not be seen...

[mack]

Your robots.txt file does not need to be displayed on your site map. In fact your robots.txt file is not really designed to be viewed by your users.

The purpose of robots.txt is to tell web crawlers (spiders) what content they are and are not allowed to index.

Your robotx.txt should be a simple text file with the correct syntax used to allow or dissallow spiders from your pages. If you do not have any content that you want disallowed from search engines then I would say leave your robots.txt out, simple dont use one.

For informatrion on writting your file you might want to do a site search for robots.txt There are quite a lot of good examples.

Mack.

[g1smd]

The default action is that visiting robots look at all HTML files in all directories of the site.

Use robots.txt to disallow certain files or folders, like intermedite steps in a shopping cart, include files, and so on.

If there is only a few files to disallow, use the meta robots noindex tag just on those few files.