lucy24 - 8:47 am on Mar 25, 2012 (gmt 0)
Well, yes, that was my point. How do you distinguish between a browser and a robot? You can't, unless the robot introduces itself as R. Daneel Olivaw or spoofs some wildly improbable browser. Don't know about your server, but mine doesn't come with x-ray vision that would let it say "This UA string may look like a perfectly respectable* MSIE 9 with the ordinary human trimmings, but it's really an evil robot from Kazakhstan".
And I'm not going to lock out those Canadians using MSIE 5 for Mac just because I think they're bonkers ;) They're my audience. Not, ahem, people who are bonkers. Canadians with elderly browsers and/or slow connections.
* For a given definition of "respectable".