Worry not, there is no risk if you do it properly. Ensure the host you are moving to is reliable, do they have a downtime agreement? I find if they offer some sort of compensation for downtime they tend to be more reliable.
Then follow Matt Cutts instructions for moving your domains across and you'll be fine.
Hey thanks for the feedback. I got the new servers set up and everything was transferred this weekend. So far everything has been great. Better than I expected, my sites have been running so smoothly and conversion are up about 3%. I guess people do like faster sites ;)
My host did a great job getting everything set up. I would recommend them to anyone. They transferred everything over in the wee hours of the morning, I have to admit I was sleeping! 0 downtime which was very nice and the Googlebot is already sniffing out the new IP.