MSNBOT and robots.txt

[Impala]

HI
I get a lot of 404's from MSNBOT ...the first thing it looks for is a robots.txt file..and since I dont have one, get the 404. I thought of putting in a robots.txt with just the following:

User-agent: *
Disallow:

That will just tell MSNBOT to crawl everything...I read somewhere that MSNBOT really likes to see a robots.txt file...is that the case? I dont want msnbot to go away after not finding the robots.txt file...!

I would appreciate some input on that. Should I have a robots.txt file at all?

I also have been thinking of disallowing the bots to crawl the cgi_bin directory, is that a good idea?

Many Thanks

[Dijkgraaf]

Yes, that would be fine for a robots.txt file.

If you are finding bots are trying to access your cgi_bin folder, then by all means disallow it.

I haven't heard of any search engine that "prefers" a site with robots.txt.

Having a robots.txt file in place will get rid of those 404 errors, so it is worth having it just for that.