On our site we have always directed visitors to the homepage when they decide to continue shopping after adding a product to their cart. I have never seen any problems with it, until last week we received an email from a customer complaining that he wasn't taken back to the product page he was last on (the one he added to the cart).
Now our site is in an industry where people usually buy a couple of items...if that helps.
I plan on doing some A-B testing on this, but wanted to know what everyone's experience with this is. I have seen sites do it many different ways.