What is the downside to having lots of one way links from directories?
The downside is if you enjoy a hand review and your backlinks are from low end directories, what's Google got to say about that?
If you're doing large volumes, there's a good chance many of those directories are neutral when it comes to ranking - so maybe no direct downside, but a complete waste of time.
It's basically impossible to get 'lots' and 'high quality' and 'niche' when it comes to directories. Remove the 'lots', and I do like directories. If they're high quality general directories (so that's what, 2 or 3?) or they're niche (so that's what, another 2 or 3?) then it seems like a good idea. After that, spend your time on something else.