I would give it a few days before you start panicking - it is most likely the new links however the drop could be because google is recalculating your link popularity - I've seen this with my site in the past - I lost ranking for keywords i was link building for but the ranking came back 4 or 5 days later and actually improved due to the links.
In the mean time you should review the links you got - what sites are they on - are the sites indexed by google? Do they have a PageRank value. I know PR is not that useful - but if the PR is 0 (IE the toolbar is white) in my experience it means that site has been penalized and if you have a link there it could be negatively affecting you.
Also, how many links did he get you and over what time period - if he got you a bunch of links over a short period of time and you haven't been link building in the past he probably did set off a flag at google.