Removing Website from Search Engines. How?

This might be the oddest question this week... but it's true!

[Dan_Vendel]

Our company have changed name due a merger and the new site with the new domain is slowly gaining traffic. We have left the old one (with same street address, phone numbers, etc but with a different name) hanging. But since they both shows for same search phrases and are listed after each other in directories, we see in the stats that people are visiting both, which must look pretty awkward: Two companies with same contact details. Doesn't add to crediility, if I say so...

Therefore: Is there any convenient way to remove the old site from all SEs and directories it's listed in?

[pageoneresults]

Hello Dan_Vendel, yes, there is one way to do this according to the specifications outlined in the various protocols.

Here's a quick outline of a current project I'm working on where we are in the exact same situation as you are.

1. Catalog all indexed pages from old domain. Make sure to run various site: searches at the top search engines to see what is indexed. Put your indexed URIs in an Excel file and sort them alphabetically (it's easier to work this way).

2. Drop a robots.txt file on the old domain and block all spiders from indexing...

User-agent: *
Disallow: /

3. Set up 301 redirects from old domain to new domain. Use your Excel file to map all relevant pages from the old to the new. Be sure to verify that the redirects are returning proper status codes in the server headers. A recursive result for a server header check would usually return first the 301 (old domain) and then the 200 (new domain).

If on Apache, use mod_rewrite (free). If on Windows, use ISAPI_Rewrite ($69.00 USD).

Our campaign just launched 2004-10-01. I'm guessing a full 60-90 days before any real results are achieved. Google will be the first to update. Yahoo! and MSN will take some time. If you follow the 301 advice above, you'll be able to capture all of that existing traffic without skipping a beat. Even though 3 months from now Yahoo! and MSN will probably have the old domain indexed, at least you've got something in place to capture all of that valuable and hard earned traffic.

Now, what do you with all of those sites where they are referencing the old URI? This is a task in itself and is something that will take some time to complete. That is why it is imperative that you implement the 301 redirects from old to new. It could be many months before those references are updated. Sidenote: There are few webmasters who check their outbound links for http status. ;)

You'll want to personally contact your top referrers and let them know of the change. A phone call may be in order.

This is just the start of it all. It will be interesting to see the end result of all this. I have a pretty good idea of how it will all pan out. I feel strongly that following the protocols is the only thing you can do. No shortcuts. Even though we have search engines who are misinterpreting the protocol, they will one day soon follow it. Better to get a head start now. ;)

[signup1]

your method does not work well. here is the best solution.

remove everything first.

write a 404 error page with 1 byte, like a space. No title, no html. Nothing except a space.

Remove robots.txt

Let google, yahoo, msn crawl the site again. They will got zero sized pages all with the same content.

BUT REMEMBER, don't use relative path, use abs, so header code 200 is sent, not 404. If SE got a 404, they will keep the old copy.

Do
ErrorDocument 404 [yoursite.com...]

Don't
ErrorDocument 404 /404.html

[jdMorgan]

Signup1 is correct that you must use a local URL-path when using Apache ErrorDocument. Otherwise, the server will return a 302-Found response instead of the desired 404-Not Found response -- See the Apache documentation for details. However, the method of simply 404ing all pages on the old domain will obviously lead to loss of the old incoming links and their associated link popularity and/or Google PageRank.

We know that Yahoo has or is still having occasional problems with handling 301s correctly, so I suppose it is a matter of whether one puts priority on making the old site disappear as soon as possible, or on trying to retain the old pages' link popularity and PR.

Personally, I'd use the 301-Moved Permanently redirect method, and take the chance that Yahoo might not handle it well or fast. As pageoneresults says, the HTTP 301-Moved Permanently server response is intended for use in precisely this situation.

I'd like to hear the reasoning behind the claim that "your method does not work well. here is the best solution."

Jim

[Red_Eagle]

Are you sure you don't want to just redirect the old site?