Alexa bot

Do I NEED robots.txt?

[TheRookie]

The alexa bot came to my site, looked at my index, then looked at my robots.txt (which doesn't exist), then left.

Is it because I don't have a robots.txt? Should I put something up there that won't change anything but will prevent this from happening again?

Thanks.

[treeline]

Bots generally regard no robots.txt as permission to do whatever they want.

So alexabot stopped by and got what it came for. It just wasn't very curious that day.

If you want bots to look at anything they want, you don't need to do anything. If seeing all the "file not found" errors for robots.txt bothers you, then create a simple file that lets them follow at will.

[TheRookie]

Heh, the file not found actually was starting to bug me, but that turned out to be a positive. I went to the searchenginewatch site and got a robots.txt to block them from my CGI and images, so I' better off than I was before.

Thanks for the reply!