You need the credit card number when you call Visa or MC to get the issuing bank's phone number, and I really doubt the bank rep would look up a transaction number to get to a customer's credit card info.
We've used this method of fraud protection successfully for many years and bank reps are always happy to provide address verification and always ask "what's the card number?"
There is no PCI issue when the credit card information is stored by the hosting company (like Yahoo). Of course Yahoo is Level 1 PCI compliant and protects that information. I don't have to be PCI compliant because i don't store or transmit any credit card info, but I DO get to see it in the Yahoo Merchant Store order manager.
Is Yahoo really the only host or ecommerce solution that provides merchants with full credit card information on transactions?
If I were selling socks or scarfs perhaps I wouldn't care so much about losing a couple dollars to someone using a stolen credit card, but I sell expensive electronics and verifying the address entered by the customer is the only way to protect my company from thieves. Of course if the AVS on the Billing address comes back as a match, then we immediately ship the product.