I've never gotten calls, but I have received email offers similar to what you are describing.
Like you, the first thing that came to my mind was this person found out about my site by clicking on my ad and costing me money!
Eventually, they'll give up. If they don't, email them or call them back and let them know you never intend to buy from them. That should make them go away.