I am getting a lot of totally irrelevant ads on some of my pages.
I did do a Google search for the keywords that should be relevant for these pages and found that there are no adwords ads even on Google for these keywords. It's a non-profit, public service sort of topic.
Do I block these advertisers or just remove the adsense banners from these pages since there are no relevant adds to show? Or do I wait and see?
You should check the terms and conditions, if Google cannot find any relevant ads for your site or if you have newly registered and they have not crawled your pages or if you have complicated code, page login requirements etc and google cannot figure out what your page content is, then they will serve these kinds of ads until everything has been sorted, i have them all over my pages.
I don't think you should ban these listings(if possible) why? they are non profit making....i assume!
but if it still annoys you, no problem, simply remove the banners from the particular pages that do not offer the Ad content you require.