Newbie - Why wouldn't I want to allow certain robots?

If I don't have anything on the site I want to keep private

[AmyG]

I just discovered the notion of robots.txt when I pulled a report from my website that said the most common error page occured when people went looking for robots.txt.

I phoned a friend who sent me info that included a link to you wonderful people.

So, I have a site that has no files or directories that I wish to keep private. I want to appear in every search engine and be as easy to find as possible for any potential customers. No shopping cart or private customer information here.

Does anyone have an example of why someone like me would not want to allow all bots?

[PatrickDeese]

Well you might want to make a robots.txt file anyhow, and just make it say:

User-agent: *

That will tell the bots that your site is open to them.

[jdMorgan]

In order to be valid, you'll need a Disallow directive. The following will tell all robots that all directories and files may be fetched:

User-agent: *
Disallow:

Include the final blank line as shown.

Placing this robots.txt file on your site will stop the 404 errors, and allow the site to be fully indexed.

Jim

[victor]

This parallel thread is an example of a bot to ban:

[webmasterworld.com...]

robots.txt only works for evil-but-polite bots.

The rude ones (whether evil or not) don't check robots.txt. For them you want stronger medicine.

The WMW robots.txt is a long, and slightly out-of-date but still useful list of polite but evil or useless bots....You'll need to use some personal discretion with it -- you may not want to ban the Google image-search bot, for example.