I'm scared too. The result is that I am extremely hesitant to touch my best site at all for fear my rankings, and thus my traffic, will fall. It's not broken, it's working, and even writing a new article changes the internal links dynamics so I hate even writing new content.
Ultimately I find myself working on my #2 and #3 sites almost exclusively to bring multiple sites up to the same success level because I just don't feel safe with only one site performing extremely well. The 2nd and 3rd sites aren't even related to the first in any way, in fact they are as different as I could muster.
I'd rather work on just one site, but I can't get wiped out in one blow. Family depends on me. That feeling wasn't as strong even 2 years ago and Panda put it over the top.
I've embraced social media with good results recently and am looking to algo proof myself further, which isn't great news for Google either.