Yes after adding breadcrumbs keywords didn't rank
I think you missed my point.
Just because you resubmit a new sitemap doesn't mean Google recrawls them immediately.
You have to check which pages were crawled vs. their keywords as I took a quick look at the site in your profile, assuming that's the site, and found category keywords ranking in the top 10 easily.
So if you have specific losses you need to correlate those to crawls while Yoast was present.
I don't think that's the problem, I think a change in the Google index algo might have played a factor here.
Happens all the time, Google spins the dials and things go South.
I wouldn't panic yet as stuff goes up and down all the time but putting Yoast in and taking Yoast out is also a bad signal as you're screwing around with your links in a major way which may make it much worse.
Unless the Yoast breadcrumbs used H!, H2, etc or something in the breadcrumbs I'm not sure what impact it could possible have.
I've never looked at those SEO plugins as I think they're all a BAD IDEA as I control my own SEO, it's not that complicated in the first place.