yes it matters. Two distinct actions are performed to access your web page from your url.
First, the full domain name is translated to an ip address via dns.
Then, your web page is requested at that ip address.
During normal browsing, these two tasks are performed sequentially. Your system first requests the ip address for the url you are requesting and then, immediately, requests the web page from that ip address.
Some spiders, for performance reasons, will disconnect the the tight temporal relationship between the dns lookup (to translate the url to the ip address) and the page request to that ip address. Google does this.
That means that Google might lookup the ip address for your domain name today, but not attempt to request your web page from that ip address until this weekend. If you have modified your nameservers to serve the site from a different ip address between those two events, google will look for your page at the wrong ip address. You can miss a crawl by changing ip addresses in the period before a crawl unless you serve your site from both ip addresses during the transition.
There is a second issue to keep in mind that could also trip you up. It may take up to as much as about 48 hours for the new dns assignments to propogate across the web. That issue involves browsers as much as spiders. This means that, for as much as two days after you assign your domain to a different ip address, requests will still executed against the old ip address.