I'll definitely agree with that. Drupal is for those who *need* the framework capabilities or advanced features. Always use the simplest CMS possible.
The killer feature on Drupal that nobody else has is Content Construction Kit [drupal.org] plus Views [drupal.org]. Joomla wasn't remotely close to that sort of out-of-the-box power last I looked. That allows me to collect data in any form and present it in virtually any form. That's why I say that for complex needs, Drupal has power beyond any other CMS I've used, way beyond Wordpress. But definitely CCK and Views are developer features and no client type of end user should have permissions for modifying those.
For me, if I need CCK+Views functionality, I use Drupal. If not, Wordpress.
Back to lorax's main point - Drupal did a lot of usability testing after releasing D6 and it was eye-opening for the community. Users habitually failed many things like simple account creation. They have made that a priority and you'll see *some* improvement for D7 (out soon), but likely not see the real fruits of that until D8. It remains to be seen at that point whether or not people have moved onto something new or not.
I do think, especially with the WPMU/WP code merge, WP is getting more powerful faster than Drupal is getting more usable, and Joomla still shoots the middle.