By refunding your money, and with no formal contract to speak of, I believe he's off the hook. Honestly don't think you have a case here. Since the domain was never signed over to you he's likely allowed to pull out by refunding you.
If I order a product online and they can't get it to me for some reason, as long as they refund my money I can't sue them. Same rules will apply here no doubt.