An interesting talk from accesibility expert Joe Clark: When accessibility is not your problem [joeclark.org].
An example given in the above article is for users of IE6 who can't resize their text - the author suggests that you can't take all the blame, you need to push some responsibility on the end user too. Accessibilty is a shared responsibility between the webmaster, the user agent (ie. browser makers), and the end user.
When do you draw the line insofar as accessibility goes? Are you doing too much?