But from a merchants point of view paying 1.5% is more than enough for me and AMEX's merchant fees are ridiculously high based on the fact that anyone who spends regularly on plastic would find it hard to get by with just an AMEX card.
Just about sums it up. But....
Depends on what countries your business comes from. Running a UK bricks & mortar business, we do not accept AmEX and do not lose business from that decision. But I would accept there may be some countries (viz this discusion above) where Amex may be the only way customers are prepared to pay.
However if I was in your position, I would cut Amex, save the 3% difference in commission between it and Visa/MC where people hve the choice.