Forum Moderators: Robert Charlton & goodroi
OUR REQUIREMENT
We want to remove one of our sites from prime visibility. In other words we basically want to zap it as far as search engine ranking is concerned. It can stay in the index if necessary, but we want to zap the traffic.
I should clearly state here that this is OUR site. There is no funny business involved at all. We have good reason to get rid of this old site, and given that just putting a one page redirect there hasn't got rid, we need other options.
SOLUTIONS?
1) I know about the Google removal tool, but we don't want to see it suddenly re-appear in 180 days. Nor do we want to trust our own competence in terms of managing robots.txt when we change servers. We need another solution.
2) I thought about deliberately invoking a penalty. The problem is, which/how?
The obvious one is font color on same color background, but I think Google forgives you when this is corrected? It's not something I have done before.
Some people have suggested that we put certain inapproriate material on there, but we are simply not prepared to stoop to anything like that.
Another option may be to play around with 301/302s to sub-domains? That does seem messy though, and again, we would probably forget about it at some point.
Any thoughts or ideas on this one?
I'm not quite sure why you don't trust yourself to manage robots.txt. If you want to block all engines, it's only two lines.
At the end of the day, whatever you do will require a certain element of management.
If the pages are for selected eyes only, you could always password protect the whole site.
TJ
However, robots.txt is undoubtedly the best solution.
Kaled.
I put up a robots.txt file with this...
User-agent: *
Disallow: /
I then used Google's removal tool. (I don't think Yahoo has one.) I submitted the request Friday evening. The site was removed from sometime yesterday afternoon.
Back to Yahoo: I resubmitted the URL to the site's home page to them. My hope is that they will check the robots.txt file if and when Slurp ever comes through.
And one last thing that I did: My site is entirely dynamic (I use ColdFusion). I've got code in the application.cfm file (a file that gets called before every request is processed) that looks at the user agent (browser type) and if it looks like a robot, I throw a 404 error and abort the request. By doing this, the spiders think the page is gone, but real users continue to see the page without any disruptions.
I hope this helps. Good luck!
If you make massive changes to the content, or just serve blank pages, Google will continue to show your old pages as Supplemental results for the next 2 or 3 years.
There is NO permanent removal method.
Maybe I didn't explain it well enough. The issue is prime visibility. Basically not ranking high on anything major. The site just sitting in a corner, albeit still in the index.
Remember the PR0 penalty? That is permanent(ish): site still there, but just not appearing on major searches.
We want something like that. Totally dampen down the traffic and visibility for good, without having to manage the robots.txt or other stuff in the future.
As I stated above, there are probably ways of attracting some sort of penalty like that, but we really don't want to do anything unethical or problematic, even though it is our site.
We just want to 'self-ban' or 'self-penalize' our site using the easiest and cleanest method.
Any ideas on this?
Totally dampen down the traffic and visibility for good, without having to manage the robots.txt or other stuff in the future.
Then you'll have to use "noindex/nofollow" as mentioned above, but to be honest robots.txt is the better solution.
We just want to 'self-ban' or 'self-penalize' our site using the easiest and cleanest method.
There's no guarantee that it will continue to work, and there's probably far more work involved in that (especially doing it in a way that doesn't impact the use of the site) than doing noindex/nofollow.
I would revisit your strategy here and do it the easy way, personally. And that's with robots.txt.
TJ
I would revisit your strategy here and do it the easy way, personally. And that's with robots.txt.
I'll agree, the robots.txt is going to be part of the solution for those that obey it. As long as that robots.txt file remains in place and doesn't change the site is not going to reappear in 180 days, not according to Google guidelines.
Have you thought about putting everything behind a password protected area? That will surely stop it from being indexed. And, it is quick and easy and will effectively remove the site from all indexes. :)
Text on same color background will attract a ban, but is it temporary (until repaired or the page changes)? I think so.
I guess I am looking for something like this but which will attract a penalty on a longer term basis. I do exclude putting offensive material on the site which I suspect would certainly work.
So it boils down to: what is the easiest way to get your own site into serious hot water, regarding a penalty?
So it boils down to: what is the easiest way to get your own site into serious hot water, regarding a penalty?
Why would you want to incur a penalty? What if you decide to undo that at some time in the future?
I'm starting to question the motives behind the questions. ;)
Text on same color background will attract a ban, but is it temporary (until repaired or the page changes)? I think so.
Just don't "repair" it afterwards. That might be easier than remembering to copy the robots.txt file to a new server. At least it's something you have to remember not to do, rather than remember to do ;-)
TJ
Whatever your reasons for disliking the robots.txt or (robots meta tag) solution, unless you want to add password protection (bad for users) or use cloaking (perhaps too complex) then it is just about your only option.
Kaled.
hmmm...
I have experimented a lot with deleting content from the search engine indexes, both with the URL removal tool [webmasterworld.com] and robot meta tags and I would advice against the two options mentioned above and use the following meta tag instead:
<meta name="robots" content="noindex,follow">
There is a huge difference with this line compared with the other two options, i.e. Googlebot keeps crawling your site. In this way it reads the robot tag every time it fetches the file and the content will be removed from the index quickly.
With the other two options Googlebot is blocked access to other parts of the site with as side-effect that those pages are not removed from the index, but go supplemental. Going supplemental is what you have to avoid in all circumstances because once a URL is labeled "supplemental" in Google it is almost impossible to remove it from the index permanently.
So if you want to remove content from the SE indexes: let the bots in and let them spider to the deepest places of your site, but tell them to not index the page content.
Meta tags is the best way to go should you not want robots.txt
Incurring a ban on your site to get it de-indexed is like driving into an oncoming car because you are our of gas.
Incurring a ban on your site to get it de-indexed is like driving into an oncoming car because you are our of gas.
It is more like driving agains an oncoming car because your brakes aren't working. It is an effective way to stop, but damage is--just as with a ban--quite severe.
The OP did not give much information about the reason of why he wants his site removed from the SE indexes, but I can think of many legitimate ones.
Copyright issues for example or a situation where the content of a site has been sold with the right of the original author to keeps his site up and running, as long as it doesn't compete with the new owner in the SERPs. Or what to think of a site which has a very strong community and doesn't need new lurkers from the search engines? Recently someone started a thread that hotlinkers from popular sites like myspace were stealing loads of bandwidth. No doubt that most of those surfers come via the search engines and going low profile can be a solution to avoid those useless visitors. Also Brett tried it some months ago when WebmasterWorld couldn't cope with the load of bots.
The idea of a self-penalty is that we don't have to worry about robots.txt, or meta-tags, or anything else. The site is zapped and low volume and that's it: clean and easy for us, for good. If you can't understand why we prefer that, I can't understand why you can't understand!
And then there are the inevetable folks who see machiavellian intent somehow. It's OUR site for goodness sake.
The REASONS aren't what this thread was about though. The thread was about the MECHANICS of what we want to do. It seems that no-one wants to discuss that though. I guess some fell on stoney ground.
The thread was about the MECHANICS of what we want to do. It seems that no-one wants to discuss that though. I guess some fell on stoney ground.
Unfortunately the MECHANICS of what you want to do is something we would probably not want to share with the general public. There are of course your numerous threads at WebmasterWorld where penalties and bans are discussed. You could easily implement any number of techniques to get your site banned. From participating in bad link exchanges to using hidden text, it is all there for the taking.
What you won't find though are the MECHANICS for what you want to do. You'll need to play the role of mechanic and assemble your toolbox based on the various topics surrounding penalties and bans. And guess what, there are literally tens of thousands of them.
For many of us, it is difficult to look at a website from that perspective. If the site is toast now, then just let it go. Don't worry about it. Sounds like it was permanently toasted. Must have been something fairly severe to end up with that status.
Have you even tried contacting google/msn/yahoo to see if they can permanently remove you from your index?
Having read both your post in december last year and yours now, I think I understand your situation. You speak the truth in that the site is still under your control now, but my strong impression that it won't be in the near future and you want it removed from the SERPs in such a way that it can't recover even if it turnes into other hands.
Unfortunately I have to disappoint you at this point. There were several posts here of people appearantly buying banned domains which they didn't know at forehand. When owner and content is totally new you can as the new owner request Google to lift the ban on that domain. So whatever your reasons are to permanently hurt the rankings of this site, after it is out of your control bans can be lifted.
