Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: goodroi
Others bombard your server with requests one after another too fast and bring your server to its knees.
Sometimes you have pseudo-sensitive data you don't want crawled.
Other times you have a development server online that you don't want crawled and indexed, just your production server should be.
I have a few. Another reason on our site is we duplicate content (with permission) from another website. That would get us penalised by google if we let them in.