The problem with this is the same as having text only versions of a site - people don't want separate because separate is rarely equal. If you can make something accessible - and it is simple once you know how - then not doing so is discrimination. And blind people are not the only ones who have issues. What about people with low vision or with cognitive disorders who may read a site and use a screen reader at the same time? There's loads of reasons to understand accessibility and to do it correctly on your main site.
People are panicking unnecessarily. Nobody's suggesting for starters that Bob Smith's personal webpage has to be perfectly coded. Nor are they suggesting that if you miss out a couple of alt attributes or accidentally forget to label one form field you can be sued. But, if you are a business and you put up a public site that a non-disabled person can use, in the same way as a brick and mortar shop, then you need to take reasonable steps to ensure that as many users as possible can complete the main call to action of the site. If it's a shop that's find the merchandise you want and make a purchase. If it's looking at photographs then it's providing decent alt text. In the same way that nobody would think it was reasonable to suggest that McDonalds had to find a way to post menus in massive text or 40 languages in every outlet - because someone who couldn't read or understand the menu can ask someone behind the till for help - a website owner should accomodate as many people as he reasonably can based on the purpose of the site. For goodness sake, why wouldn't you want to accomodate as many people as you can? It means more business and goodwill in the community.
Plus, as keeps being pointed out, why deny a massive minority group the ability to shop, pay bills, bank, etc., using the internet without help, why deny them that independence which they haven't had in the past *because* they haven't had access to something like the web when it takes so little time and effort to build accessible websites? How would you feel if someone said to you, "I know I could have built my site so you can buy your groceries without needing help, but I couldn't be bothered learning how so instead you'll have to rely on a friend/relative/carer to do it for you. It must feel good having to rely on others when at your fingertips is a way to be able to do it yourself."