Should I expect to see drastic negative movement of my site in google?
or as long as it can get to www.domain.com, it doesnt care about the ip or host.
Changing hosts for a domain doesn't affect Google rankings long term, sometimes it takes a little bit for them to catch up with changes because of DNS update frequency. They are better than other search engines from what I've always seen - others still seem to take longer.
With moving from a static IP to another static IP you could remove the pages from the old once it's propogated to the new all over (have people in other locations check for you)and possibly use a 301 redirect. With moving from a shared IP to another it's not quite as smooth.
You might want to put up a link to your site using the new IP number for the link instead of the domain name - that can sometimes help expedite things, especially for those moving from shared to static.
Don't worry if it takes a bit, it does straighten out even when there's a slight delay but it should be fine.