Forum Moderators: open
1. Can the 301 be written for the folder, i.e. the entire site?
2. If the 301 can be written for the entire site and not individual pages, does it matter if the site at the new location changes content, but not file names?
3. Does the old site have to be kept at the old location for a period?
4. What is the best use of a robots.txt file for this?
5. Any other issues?. Thanks and more thanks, especially to Brett who has mentioned some of this before.
If the URL did not change, all you should have to do is get your DNS records updated. Contact
your domain name registrar if you have any problems.
In case you didn't describe the problem quite right, the answer to your question - assuming the URL
DID change and that your are hosted on an Apache server - is that you can use a simple one-line
directive in .htaccess at the old site to redirect all resource requests to the new URL.
This is of the form:
RedirectPermanent / [newdomain.tld...]
for the "whole site"
or
RedirectPermanent /subdir [newdomain.tld...]
See the Apache server documentation for mod_alias at www.apache.org
If none of this sounds right, please fill us in on what server you're running, and why the
DNS update didn't work.
Good luck,
Jim
P.S. If you're e-mail address is any indication, I'm a customer. :)
Yes, if you're just changing DNS, keep a copy of the site on the old server long enough for the
new DNS records to propagate world-wide. I've heard 4-5 days, but keeping it longer won't hurt.
And again, assuming you're just changing DNS, you shouldn't have to mess with a redirect or with
robots.txt.
Things only really get complicated when you are changing domain names...
Jim
Some postings have touched on these topics, but clarity would be of great use to us, because we have migrated 4 sites to dedicated servers (same URL, different IP's) and we have lost ALL presence on all 4 of them. A correct 301 solution may help save the day.
I don't think there is a correct 301 solution here. You've got one of two problems:
First possible problem:
You've moved 4 domains to 4 separate IP addresses, and Googlebot is still visiting the old shared IP, because its DNS cache still associates the domain names with the old IP address.
If you set up a 301 redirect leading from the old server to a domain name, you'll be redirecting from the original server to the original server. Looping is pointless, and possibly something that will scare away search engine robots for good.
If you set up a 301 redirect leading to the new IP addresses, you risk getting the new servers indexed by IP address instead of domain name. (Unless Googlebot does nslookups on raw IP addresses, which could solve the entire problem, but I wouldn't count on that.) You could end up with the content indexed at the domain name and the IP address. Might cause duplicate content problems.
(Googlebot servers are known to cache IP number/domain name pairs considerably longer than DNS servers do; this is not the normal "wait 2 days for DNS propagation".)
If the problem turns out to be that Google is still looking at the old IP address for the relocated domains, your safest option is to maintain the content on both servers until you're sure Google is grabbing pages from the new server.
Second possible problem:
It's something else. No way to know if you won't tell us what Googlebot is doing.
jdMorgan's advice (4-5 days) is good for browsers and ISP proxy caches, but search engines cache DNS for much longer.
As mbauser points out, HTTP redirection is not the answer. Maintaining the content on both IP addresses a a good solution, I'd say for a few weeks. Google
The other option, which I used recently, is to buy port redirection from the old provider. Requests for the old IPs are transparently directed to the new IPs. This worked nicely for a few hundred sites we had to move a couple of months ago.
Yeah, I initially missed the point about the IP-based requests.
I was going to suggest an IP-based redirection - redirect the old IP (not domain) to the new IP.
I figured that since the bot came in using an IP address, it might already know it was in "IP mode"
versus domain mode. But this depends on how smart the interaction between their domain lists
and their DNS caching is. You certainly don't want to create a recursive "lookup loop" as mbauser2
points out, by redirecting by IP address to the new domain.
The port redirection sounds like the way to go. I hope relations are still good w/the old hosting
service! Keep it in place until Googlebot no longer shows up in the logs on the old server, and
until your listings in the Google search results are stable through at least one complete update
cycle.
Jim
By the way, we do not have access to our stats as the Hosting company maintains they were not transferable to the stats tool when we moved. I am asking them for archival material, but it is not forthcoming yet. So the info on Googlebot is not available.
Port direction might be something the hosting company does not want to pursue at this time, so we may be left with that 301 solution.
Comments and thanks to
mbauser2
Once the new DNS has propagated, how would the 301 direct from the old IP address to the new IP address be recursive? When Googlebot returns to the old IP adddress, the direct should be to the new one. In any event, though, would there be a chance of a mirroring penalty by Google with duplicate content up on the two addresses? What would be the danger of linking to the IP address instead of the domain name? And then there are the anecdotes about Google flipping back and forth between the two sites over the course of two or three indexes...
jdmorgan
You mentioned that interaction between the hosting company's domain lists and and their DNS caching is real important.. what should I look out for, specifically in that regard, if they have the server setup to make it work?
Ciml
If we can wrangle port direction, are there any spidering issues...when would Google pick up the new address and know to drop the old? Any problems with getting rid of the old address... do we simply just drop the old IP address when the new index picks up the new IP address? If we drop the old address, will Google still have both in their index, with a resulting series of 404's when we drop the old IP address?
Lastly, any comment on this statement from a Google engineer on the subject [searchengineworld.com...] recommending 301's as a solution?
Don't mean any of this to be a sub-career for any of you.. I sure thank you for helping out a beginner. I'm grateful for any last response.
By the way, we do not have access to our stats as the Hosting company maintains they were not transferable to the stats tool when we moved. I am asking them for archival material, but it is not forthcoming yet. So the info on Googlebot is not available.
Stats tools are crap. Here's my hard-and-fast, completely unbreakable rule of web hosts: If they don't give the raw access logs, you can't trust them. Stat "tools" are vunerable to all sorts of idiotic configuration errors by half-witted techs. Get away from them and sign up with a real hosting company.
Unless you can me a yes-or-no answer to the question "Is Googlebot still requesting your domains from the old IP address?", I can't help you much.
Once the new DNS has propagated, how would the 301 direct from the old IP address to the new IP address be recursive? When Googlebot returns to the old IP adddress, the direct should be to the new one.
Bzzt. Wrong. Answered in my first post. You need to learn more about DNS and virtual hosting. (Asking me questions I've already answered makes me cranky.)
Repeating: Googlebot caches number/domain pairs at the Googlebot server, not its local DNS server, and doesn't repoll as often as normal DNS servers.
Showing Googlebot your domain name again won't magically make Googlebot do another DNS lookup. (Even normal DNS systems don't do that. It would slow down the Web something terrible.) Googlebot will just use the IP number it already has cached for that domain.
I figured that since the bot came in using an IP address, it might already know it was in "IP mode"
versus domain mode. But this depends on how smart the interaction between their domain lists
and their DNS caching is.
Careful with the distinction between "IP mode" and "domain mode" when talking about virtual hosting (as in 4 sites on one IP number). Agents have to request virtual domains using the IP number and the domain name. (The agent connects to the IP address, and sends a "Host" header along with the HTTP request. If it doesn't include the "Host" header, the server doesn't know which domain to show.) If Googlebot has sucessfully requested a virtual domain already, that means it's already associated the domain name with an IP number.
I don't know if discovering a new IP address forces Googlebot to do a name lookup or not. Running some test searches through Google, I came up with this bizarre search result [google.com], where searching for the URL-with-domain-name produced a result that links to the URL-with-IP-number. I don't know what to make of that. I think I'm going to sleep on it.
You need to think about the difference between Google's DNS servers keeping track of IP addresses, and Googlebot keeping track of Web adresses. The port forwarding would be completely transparent to Google. If you drop the forwarding before Google catches up with the DNS change then you could still suffer.
mbauser2, don't worry about the bizarre search result. That's what happens when Google spiders both addresses and keeps just one.
