Why do you need robots.txt?

[gordele]

Hello guys,

I was just reviewing my "Error Log Messages" and I saw that this file "robots.txt" caused an error. My question is, what is the robots.txt and why is it missing from my site? Im almost sure that this file has something to do for the search engine crwalers but Im not sure why I dont have one and If I should add one?

Thank you guys as always for your help :)

[DanA]

You need one if you want to limit visits of robots, crawlers...
The well behaved robots, crawlers... ask for robots.txt before crawling the site.
Lots of information there :
[webmasterworld.com...]

[Jackblack]

why ould you want to limit the amount of robots that come to your site? Isnt that a good thing?

[treeline]

Usually you want lots of googlebot and other robots. But you may develop a situation where you don't want them on certain pages. For example, I exclude them from add_url pages in a directory. Since the search engines don't index these pages, it makes it hard for spammers to find and add junk links. Otherwise I love robots.

If a robot is using toooo much bandwidth with frequent visits, and not sending you any visitors you may want to ban it. Some people have done this with MSN.

Another good reason to have a robots.txt file, even if it allows all robots, is to clean up your error log of all those file-not-found errors.

[Lord Majestic]

Why ould you want to limit the amount of robots that come to your site? Isnt that a good thing?

It depends on your point of view. If you have short-term desire to minimise your own bandwidth costs (not applicable to most people as they don't use up typical allowance) then you might want to cut out those search engines that don't drive traffic to you straight away, ie allow G/M/Y and cut off everyone else.

If you have a long term view then you would want to ensure that your site's future is not wholly dependent on search engine oligopoly, and in some instances monopoly of one search engine. This also adds up as acting in good spirit that allowed Internet become what it is now (or at least what it was yesterday)

Its chicken-egg situation - Google did not become popular all of a sudden, and sites who were crawled by them did not get traffic straight away.

If your bandwidth costs are fixed and new bots behave well then it is wise not to block them. You may not win jackpot straight away and 99% of them will be useless, but 1% will bring you pay back.