fathom - 4:39 pm on Jun 1, 2013 (gmt 0)
The existence of unnatural links penalties proves beyond any shadow of a doubt that Google can penalise inbound links (whether you placed these links or not yourself). There is no such thing as a "slight rise, slight fall" when your site is penalised for unnatural links. The question REALLY is : do Google always notify us when they penalise us for links pointing to our site? We can never know that, but we can observe.
Unfortunately, casual observations in volume does not qualify as proof of anything. It merely say "something happened". You can't do SEO that way and expect to gain any sense of value out of your casual trappings. Using your own words, you have nothing BUT shadows of a doubt.
But I will bow out of this decision as well and concede to your superior SEO intellect.