Robots.txt - Why?

[warlordbb]

Other than keeping SE bots out of areas that you don't want indexed (which I don't see as being absolutely vital for us either), is robots.txt really worth worrying about for the average site?

IOW, aren't the "bad" bots going to do bad things even though you have a file that tells them not to?

I'm sure that there are "bad" bots out there that will respect a robots.txt file, but I would think the vast majority of "bad" bots won't.

I've got my hands full trying to manage SEO stuff, I'm tempted to stop worrying about robots.txt and just remove it from our site or put an allow everything.

[john_k]

Whether it is worthwhile to prevent the "good" bots from spidering areas of your site is your own call. If it is worthwhile, then robots.txt is an easy way to do it. If it's not worthwhile, then don't bother.

[jdMorgan]

A few reasons:

1) Prevent 404-Not Found entries in log from robots looking for robots.txt
2) Control bandwidth costs by limiting spidering.
3) Control presentation in various SERPs by controlling spidering.
4) Identify good vs. bad bots:

Disallow a 'page' in robots.txt

Rewrite that 'page' URI to a script.

Use that script to ban any bad-bot that fetches the Disallowed 'page' URL.

Jim