I think they can be a major usability issue, particularly when it comes to email and other devices where long URIs can become a problem.
Yes, the underscore is an issue in links. Print a page that has raw links with underscores and what do you see? A lot of links with what appears to be spaces and not underscores.
Using underscores also creates one more keystroke for the user, the shift key. When I say user in this instance, I'm referring mostly to the developer/maintainer of the site.
Let's define Fancy URI. If it's more than a certain character length, then it is too fancy. If it contains too many delimiters, it is too fancy. If it doesn't properly describe the resource in a short and succinct manner, it is too fancy. If it contains a file extension, you may not be thinking ahead.
I've just found that working with clean URIs make it so much easier to manage from a variety of standpoints. From usability at the consumer level to usability at the programming level. Everybody wins.