Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
IOW, aren't the "bad" bots going to do bad things even though you have a file that tells them not to?
I'm sure that there are "bad" bots out there that will respect a robots.txt file, but I would think the vast majority of "bad" bots won't.
I've got my hands full trying to manage SEO stuff, I'm tempted to stop worrying about robots.txt and just remove it from our site or put an allow everything.
1) Prevent 404-Not Found entries in log from robots looking for robots.txt
2) Control bandwidth costs by limiting spidering.
3) Control presentation in various SERPs by controlling spidering.
4) Identify good vs. bad bots: