I removed the sitemap on my best site nearly a year ago and did not suffer ANY noticeable ranking loss. The site, being older and well established, is crawled regularly and the sitemap was not providing any real benefit imo.
By removing it I have realized a performance benefit since it is no longer drawing resources from my database.
I can't say that I noticed any change in Bing/Yahoo either though I didn't watch them as closely. I gather stats from other sources so I didn't need the information provided by GWT but it is useful if you have to internal tracking.