The most difficult thing is to find the exact cause of keyword ranking drop. I am pretty sure that its not Back Links, i guess google will devalues the links which comes from directories, forum or yahoo answers, rather than taking an action.
The only thing that can go bad is the content of the site. Generally my content writing starts from getting most of the data from google search only, then i would edit the relevant data, write in my words and also adding own points also. I guess it is acceptable as many would follow the same approach for content development.
Also the worrying point is my competitors approach, i.e copying content from other sites, creating pages with similar content with just changing a specific state or country. Still there is no action from google on that site. The only difference is their site is 5 years old and mine is 1 only.