Forum Moderators: open
6. Kind of redundant but, stopping nefarious googlebot spoofers from hammering away at your server
A better way to phrase #6 IMO would be to "Stop SE hijackings of pages on your site" as more often that not I've found it's really the SE being directed to crawl through a proxy server.
Item 5 should be worded more strongly or another item added. Script kiddies ain't too much hassle and fairly easily blocked - one false move and they're toast. The major problem is botnets trying to plant SQL Injection and suchlike exploits on your sites. Although at the moment they seem to be off attacking something else - Georgia, again?
My only reason for blocking bots: Stealing is wrong.
Not sure if this reason is a big one: stop sites pretending to have information about your site in order to draw traffic to themselves. For example, aboutus.org has absolutely no information about our domains except the name but claims it has. One of them has never been (legitimately) indexed by any search engine - it's disallowed in robots.txt - and when you try to go to the page it's completely blank. Personally I blame google for letting them get away with this: I see several other similar scams when googling for stuff. It's another of those scams that ebay and kelkoo (used to?) perpetrate: "Buy DSTILES Now" - I KNOW I'm not for sale!
Others "services" to block include nebuadd, phorm, barefruit and other site interceptors who usually download non-compressed data to make their own life easier, hence not only pushing up the bandwidth but making advertising revenue on the side. Unfortunately these hits can't be intercepted without a lot of hassle and relying on such things as javascript, which is (should be) taking a down-turn at present anyway because of security (SQL Injection infected sites etc).
Not sure about some of the "security" services, either, some of which give a single-point data source to hackers. I'm not really keen on the world knowing what OS my server is running and how fast my pages load and punters can discover that for themselves by legitimate browsing.
