tschild - 1:13 pm on Jul 13, 2003 (gmt 0)

The lines you quote are not a robots.txt file but meta tags in the HTML head.

If you want to have robots index everything you do not need any <meta name=robots" ... lines at all.

Regarding the robots.txt file (which is not part of your web pages but a separate file at your domain root), if you do not want to disallow spiders from everything you could e.g.

a) put an empty robots.txt file (http://www.example.com/robots.txt) at your domain root

b) create a robots.txt file referencing a directory that does not exist, e.g.

User-agent: * Disallow: /ficticious-directory/

Advantage of this: you can build on this syntax if you later do want to keep spiders from some parts of your site.

The worst alternative is not to have a robots.txt file because
1) this swamps the real errors in your error log
2) some webspaces are set up to serve the default page (rather than an error page) as answer to a request for a file that does not exist. I have seen quite a few sites where a request for robots.txt returns the home page of the site - a search engine spider programmed in a non-robust way might choke on that.