Serving 404 Not Found from a Different Domain

[steaprok]

We recently did an enterprise site migration, where we merged 4 large sites into one existing site.

Each of these sites had their own rankings, link profiles, etc.

My main question revolves around how to handle the 404's on the old sites.

The dev team is insisting on creating a catch all 404 on site A (old site), which 301 redirects to a 404 on site B (new site) .

I oppose this for multiple reasons. The first being that I think they should skip the 301 of the 404, and just serve a 404 from the originating site, with s small blurb about the migration and links to the new site vs what they are recommending, which is to just create a catch all 404 on the old site, that will 301 redirect everyone to the new 404 page on the new site

Any thoughts on this? How would you handle?

Anyone ever served a 404 from a different domain? seems like it would cause more confusion with the overall migration and users. thoughts?

All sites are on different servers.

Thanks

[Robert Charlton]

I oppose this for multiple reasons. The first being that I think they should skip the 301 of the 404, and just serve a 404 from the originating site

You are absolutely correct. It's extremely important that the user agent (like Googlebot) receive a message that the requested url was a 404, though, and redirecting prevents that from happening. By not serving the 404, you will end up creating dupe content issues for the requested pages that never should have happened.

Here's our best discussion on the topic, and rather than try to reproduce it, I'll refer you to it, and let others jump in with further comments....

Custom Error Pages - Beware the Server Header Status Code
http://www.webmasterworld.com/google/3626149.htm [webmasterworld.com]

It is completely possible to create a dedicated 404 error page in Apache... one which will be user friendly, and return the 404 status code for the requested url. It's easy to do.

By this time, I'm sure some in IIS have figured it out as well... but many old school IT guys (in both Apache and IIS) just plain have it wrong.

PS: You add a twist, btw, which is that not all urls may have been redirected to the new domain. I feel that it's important that 404 reflect the query, so a catch all 404 page, IMO, wouldn't work.

Best way to handle these probably involves how you've handled your redirects, and that will take some taking through. I'll leave that for others more experienced. I myself would merge the sites one at a time, rather than four all at once.

[edited by: Robert_Charlton at 2:26 am (utc) on Jul 2, 2014]

[phranque]

I think they should skip the 301 of the 404, and just serve a 404 from the originating site, with s small blurb about the migration and links to the new site vs what they are recommending

you've got it right.

what they are recommending is seen by google as a "soft 404".

http://support.google.com/webmasters/answer/181708 [support.google.com]:

Returning a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404) can be problematic. Firstly, it tells search engines that there’s a real page at that URL. As a result, that URL may be crawled and its content indexed. Because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently and your site’s crawl coverage may be impacted (also, you probably don’t want your site to rank well for the search query [File not found]).

[buckworks]

My preference would be to use a 301 redirect whenever you've got something sensible to redirect the old URL to.

Whenever you can, 301 redirect old URLs to the nearest equivalent content pages on the new site. That will salvage Page Rank if the old URLs had any inbound links, and it's also a better experience for users.

Ideally you want a single step 301 redirect, no chains of redirects.

Depending on how the old and new URLs are formed, it might be possible to do a lot of redirection with regular expressions.

Build them by hand if you have to, however, at least for the old pages that had inbound links.

Only let an old URL go 404 if there are no redirect possibilities that make sense.

If the content is truly gone, use 410.

[steaprok]

wow, thanks for all the quick replies. very much appreciated.

and thanks for the links as well, now I have something else to support my argument with the table of "marketing execs" thinking they have the magic pill that will solve all of our problems.

[lucy24]

Wait, wait, come back, there's another aspect :)

My main question revolves around how to handle the 404's on the old sites.

If you've moved an entire site, there are no 404s because the entire thing is gone.

If you're blanket-redirecting all requests from old site to new site, it's more trouble than it's worth to keep track of where the request originated. A 404 is a 404 in that case. Wasn't on the old site, isn't on the new one.

The only way it even remotely makes sense is if you're redirecting only specific, named pages. If there's anything you've chosen not to redirect, that's more appropriately a 410-- whether it was already deleted, or just didn't make the cut when it came time to move. Meanwhile, if something isn't getting redirected because it never existed in the first place, why get the new site involved at all?

[Robert Charlton]

If you've moved an entire site, there are no 404s because the entire thing is gone.

lucy... this covers the "another twist" PS I'd brought up, but not with such clarity as in your explanation. If a blanket-redirect, then there's no concern... and otherwise these were urls that didn't matter or didn't exist.

Sounds like the mindset of the dev team is that they wanted no possibility of disappointment, which is what I was wanting to consider. I think that would mean maintaining custom 404/410 pages for each of the old domains... something that IMO would be overkill.

[aakk9999]

The best to do is to configure all four domains to point to the same webspace on the server. So the same script (or the same .htaccess) would handle requests for all four websites.

In this case a script (or .htaccess) can inspect the domain name and do the appropriate thing (redirect or return 404 straight away).