Well, when I spend time on the phone with a client, I call that a "meeting". Consequently it's time spent on their project, and that's billable.
Your mileage may vary.
When you call up a retailer to ask about an item they sell, they'll spend time with you. That costs time, hence, money. But they don't add that to the price of the item, do they. It's absorbed as part of the cost of running a business.
So you should differentiate between time spent discussing and planning a project, and time spent managing your business. Plus, it's rude to surprise a client with an invoice for something they didn't know was billable. If you're going to charge for phone conversations, make sure the client knows that before they dial you up, or tell them so before you hang up.
I sense that you're annoyed because you don't like spending hours on the phone. I can relate. You've got to decide whether customer service (if you wanna call it that) is part of your business, or part of a billable project. Make sure you aren't inventing reasons to invoice the client to punish them because you find them annoying.