Personally, I think sitemaps.xml files are EXTREMELY overrated and often misused. A typical web site if well architected with good internal linking structures (top navs, left navs, breadcrumbs, footers, etc.) should never require a sitemap.xml file, even one with thousands of URLs. Most webmasters are simply wasting their time with sitemap.xml files.
IMO there are only a couple of situations where a sitemap.xml is needed:
1) If your site has crawlability issues that would prevent the engines from discovering all of your important URLs during the normal crawl process and
2) If your site has 100s of thousands or millions of URLs.
In scenario 1 above, a sitemap.xml can actually help get URLs indexed that might not otherwise have been discovered during a normal crawl of the site.
In scenario 2 above, a sitemap.xml does let you prioritize which URLs you want the engines to index from your site using the <priority> element. You can basically tell them, "Index this set of URLs first. If you still have room in your index for more of my site's URLs then index this set of URLs next. And if your still have room in your index for more of my site's URLs then index this set of URLs next..." and so on.
Other than those two scenarios, I think they are truly a waste. I think it's funny how people will throw up a brand new site and be so worried about getting a sitemap.xml created for it so that they can submit it and get indexed. Why bother? You might get it indexed by submitting a sitemap.xml, but it's not going to rank without any kind of links. In fact it could be de-indexed a few days or weeks later if it doesn't acquire any inbound links. Getting a new site indexed with a sitemap.xml is pretty much worthless. That time could be better spent building links and waiting on the engines to discover and index your site the old fashion way - by following links.
Google's recommendation that you use them is just a "party line"... kind of like your car dealer telling you that you should get your oil changed every 3750 miles.
Sitemaps do not get your site crawled more often, they don't get more URLs indexed (unless your site has crawlability issues), and they won't help your rankings. Even Google tells you they are used to "assist" in crawling. But they are not even bound to following the priorities you might set in the sitemap.xml. If inbound links exist to one of your lowest priority URLs, they will likely still index that lowest priority URL before a higher priority URL with no inbound links.
So unless your site falls into one of the 2 scenarios above (one with crawlability issues or a huge site), removing your sitemap.xml should not negatively impact your site's rankings.