jkwilson78 - remember your old site is not in the vacuum. Your neglected site may have had a bunch of high quality links (high quality link does not mean it has perfect anchor text) that built trust to the site, and competition might have been basing on crap links only. So you did not go up but the competitors could have gone down :)
From my experience the best links rarely have the anchor text I want. Usually they are non-anchor links or just domain name links or feature not very targetted anchor text. But I am no expert at all ;-)
Again, at the beginning Google made their SERPs too powerful and have been fighting to make them less powerful for years. By powerful I mean that being #1 for your keyword was the holy grail. The fight with SERPs power can be seen by more and more ads, universal search, etc. so companies spend less on SEO and more on advertising.
All these changes try to make SEO even more random business. For a small webmaster like me, anchor text removal is a good idea. I can spend more money on good content and on-page optimization then spamming people for links.