Xoc - 4:36 pm on Jan 25, 2002 (gmt 0)

One reason to have a short robots.txt is that it may actually take less bandwidth than the 404 page that you send when it doesn't find it. The web spider asks for the robots.txt. When it doesn't find it, your web server sends a 404 error, that actually has a web page attached with error text. That 404 eats bandwidth. It also takes more server processing time, I think, to process a 404 than to send a file that is actually there.

So you are probably better off just having a blank robots.txt than not having one at all.