We got screwed both ways. We had a bunch of sites that were anywhere from #1 for 2yrs and mostly top 8 sites in many areas. Joined site match in hopes of maintaining positions, ALL sites disappeared almost instantly. We cancelled site match and NONE of the sites have returned to index must less prior positions. Go figure.
I quit using site match arround January of 05. At the time I was spending around $7 per day. Nothing too big but felt that I should get the same results naturally so I canceled. Around March I had a significant increase in rankings. I mean I owned maybe 20 competitive keyword phrases. I did due massive link campaigns around the end of 04 though. Well anyways with the last update (August) all my rankings dropped to around 3rd to 5th.
IMO it wont hurt you too much to cancel. I even called yahoo to make sure that I wouldnt get dropped. They assured me and I canceled.