beren - 11:36 pm on Dec 29, 2006 (gmt 0)

This might be a dumb question, but I did not see one like it anywhere, including the forum library.

Between my cliets' sites and my own, I oversee about 50-80 websites. None are more than 200 pages. None have a robots.txt file on the server. I always thought robots.txt was for very large sites or when site owners wanted to control what the engines spidered. We have nothing to hide on any of our sites, so have never used them. As far as I am concerned, anyone (including robots) can look at the sites.

I'm now adding XML sitemaps because the major engines say they prefer that we do so. I notice that in Google Webmaster Tools, when logged in, in the Diagnostic area there is a link for robots.txt analysis. It says Last Downloaded and has today's date. Under status it says 404 (Not found). Which makes sense because there isn't one. Google then says: "We check for a new robots.txt file approximately once per day."

Is this like XML sitemaps where the search engines prefer that we have a robots.txt file? I'll put them in if the major engines really want me to, but I see no particular reason to otherwise.