Said that, if you change your TTL to a very short period, like 1 minute. Then wait, make no changes for the period of your previous TTL value (to let it expire). Then change the NS servers. Technically you should have no downtime, or not more than 1 minute anywyas. The new DNS servers should have longer TTL (not 1 minute).
Note: If you have an extremely busy site, then this might be a problem, because your DNS server will be hit way too often and may crash.