Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
Others bombard your server with requests one after another too fast and bring your server to its knees.
Sometimes you have pseudo-sensitive data you don't want crawled.
Other times you have a development server online that you don't want crawled and indexed, just your production server should be.