Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Do IPs matter during crawling?

11:23 pm on Apr 9, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member korkus2000 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 20, 2002
votes: 0

Say I reroute my domain name and there isn't a second of down time. It just changes DNS to a server that has a different IP. Does googlebot get confused by this? Does the IP really matter or is it all just based on the actual domain name?
12:14 am on Apr 10, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 18, 2002
votes: 0

yes it matters. Two distinct actions are performed to access your web page from your url.

First, the full domain name is translated to an ip address via dns.

Then, your web page is requested at that ip address.

During normal browsing, these two tasks are performed sequentially. Your system first requests the ip address for the url you are requesting and then, immediately, requests the web page from that ip address.

Some spiders, for performance reasons, will disconnect the the tight temporal relationship between the dns lookup (to translate the url to the ip address) and the page request to that ip address. Google does this.

That means that Google might lookup the ip address for your domain name today, but not attempt to request your web page from that ip address until this weekend. If you have modified your nameservers to serve the site from a different ip address between those two events, google will look for your page at the wrong ip address. You can miss a crawl by changing ip addresses in the period before a crawl unless you serve your site from both ip addresses during the transition.

There is a second issue to keep in mind that could also trip you up. It may take up to as much as about 48 hours for the new dns assignments to propogate across the web. That issue involves browsers as much as spiders. This means that, for as much as two days after you assign your domain to a different ip address, requests will still executed against the old ip address.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members