I don't know for sure if the removal of my G sitemap.xml has helped me in the SERPs -- But something has happened. That's for sure.
It could also be that it just was about time for Google to release my website from the 'Sandbox'/trust filtering. Maybe?
But as mentioned, I have read before that other people have experienced changes after removing their XML-sitemap from GWT.