Why don't you just replace whatever pages are there with blank pages, and THEN do the removal thing?
You can add "nofollow/noindex" tags to pages.
I'm not quite sure why you don't trust yourself to manage robots.txt. If you want to block all engines, it's only two lines.
At the end of the day, whatever you do will require a certain element of management.
If the pages are for selected eyes only, you could always password protect the whole site.
If the site is entirely dynamic, you could add a robots noindex meta tag to every page.
However, robots.txt is undoubtedly the best solution.
Except that there *are* spiders that don't follow it, whether by accident or design.
I just went through this with one of my sites. For a number of reasons, I wanted it completed out of both Google's and Yahoo's indexes. Here's what I did...
I put up a robots.txt file with this...
I then used Google's removal tool. (I don't think Yahoo has one.) I submitted the request Friday evening. The site was removed from sometime yesterday afternoon.
Back to Yahoo: I resubmitted the URL to the site's home page to them. My hope is that they will check the robots.txt file if and when Slurp ever comes through.
And one last thing that I did: My site is entirely dynamic (I use ColdFusion). I've got code in the application.cfm file (a file that gets called before every request is processed) that looks at the user agent (browser type) and if it looks like a robot, I throw a 404 error and abort the request. By doing this, the spiders think the page is gone, but real users continue to see the page without any disruptions.
I hope this helps. Good luck!
Google will add the site back into their listings after 90 or 180 days. They will do that even if the site does not exist, or domain has expired.
If you make massive changes to the content, or just serve blank pages, Google will continue to show your old pages as Supplemental results for the next 2 or 3 years.
There is NO permanent removal method.
"There is NO permanent removal method."
Maybe I didn't explain it well enough. The issue is prime visibility. Basically not ranking high on anything major. The site just sitting in a corner, albeit still in the index.
Remember the PR0 penalty? That is permanent(ish): site still there, but just not appearing on major searches.
We want something like that. Totally dampen down the traffic and visibility for good, without having to manage the robots.txt or other stuff in the future.
As I stated above, there are probably ways of attracting some sort of penalty like that, but we really don't want to do anything unethical or problematic, even though it is our site.
We just want to 'self-ban' or 'self-penalize' our site using the easiest and cleanest method.
Any ideas on this?
|Totally dampen down the traffic and visibility for good, without having to manage the robots.txt or other stuff in the future. |
Then you'll have to use "noindex/nofollow" as mentioned above, but to be honest robots.txt is the better solution.
|We just want to 'self-ban' or 'self-penalize' our site using the easiest and cleanest method. |
There's no guarantee that it will continue to work, and there's probably far more work involved in that (especially doing it in a way that doesn't impact the use of the site) than doing noindex/nofollow.
I would revisit your strategy here and do it the easy way, personally. And that's with robots.txt.
|I would revisit your strategy here and do it the easy way, personally. And that's with robots.txt. |
I'll agree, the robots.txt is going to be part of the solution for those that obey it. As long as that robots.txt file remains in place and doesn't change the site is not going to reappear in 180 days, not according to Google guidelines.
Have you thought about putting everything behind a password protected area? That will surely stop it from being indexed. And, it is quick and easy and will effectively remove the site from all indexes. :)
We have thought long and hard about this, and robots.txt is not the route we want, for several reasons, some of which are internal to us. Hence, the question regarding 'self penalty'.
Text on same color background will attract a ban, but is it temporary (until repaired or the page changes)? I think so.
I guess I am looking for something like this but which will attract a penalty on a longer term basis. I do exclude putting offensive material on the site which I suspect would certainly work.
So it boils down to: what is the easiest way to get your own site into serious hot water, regarding a penalty?
|So it boils down to: what is the easiest way to get your own site into serious hot water, regarding a penalty? |
Why would you want to incur a penalty? What if you decide to undo that at some time in the future?
I'm starting to question the motives behind the questions. ;)
|Text on same color background will attract a ban, but is it temporary (until repaired or the page changes)? I think so. |
Just don't "repair" it afterwards. That might be easier than remembering to copy the robots.txt file to a new server. At least it's something you have to remember not to do, rather than remember to do ;-)
With respect, the penalty idea is ludicrous.
Whatever your reasons for disliking the robots.txt or (robots meta tag) solution, unless you want to add password protection (bad for users) or use cloaking (perhaps too complex) then it is just about your only option.
So the question is - What is the easiest way to do some serious long term damage to a site, that can't be repaired quickly?
(I'm trying to think of a hypothetical situation where one would want to do this to one's own site, within the parameters as thus far given, and I'm drawing a complete blank)
Some have suggested robots.txt and others adding the robots "noindex,nofollow" meta tag in the header.
I have experimented a lot with deleting content from the search engine indexes, both with the URL removal tool [webmasterworld.com] and robot meta tags and I would advice against the two options mentioned above and use the following meta tag instead:
<meta name="robots" content="noindex,follow">
There is a huge difference with this line compared with the other two options, i.e. Googlebot keeps crawling your site. In this way it reads the robot tag every time it fetches the file and the content will be removed from the index quickly.
With the other two options Googlebot is blocked access to other parts of the site with as side-effect that those pages are not removed from the index, but go supplemental. Going supplemental is what you have to avoid in all circumstances because once a URL is labeled "supplemental" in Google it is almost impossible to remove it from the index permanently.
So if you want to remove content from the SE indexes: let the bots in and let them spider to the deepest places of your site, but tell them to not index the page content.
I would agree with Lambert here. The Google tool does not work correctly. Most pages simply are removed from the visible index then show up about 4-6 months later.
Meta tags is the best way to go should you not want robots.txt
Incurring a ban on your site to get it de-indexed is like driving into an oncoming car because you are our of gas.
|Incurring a ban on your site to get it de-indexed is like driving into an oncoming car because you are our of gas. |
It is more like driving agains an oncoming car because your brakes aren't working. It is an effective way to stop, but damage is--just as with a ban--quite severe.
The OP did not give much information about the reason of why he wants his site removed from the SE indexes, but I can think of many legitimate ones.
Copyright issues for example or a situation where the content of a site has been sold with the right of the original author to keeps his site up and running, as long as it doesn't compete with the new owner in the SERPs. Or what to think of a site which has a very strong community and doesn't need new lurkers from the search engines? Recently someone started a thread that hotlinkers from popular sites like myspace were stealing loads of bandwidth. No doubt that most of those surfers come via the search engines and going low profile can be a solution to avoid those useless visitors. Also Brett tried it some months ago when WebmasterWorld couldn't cope with the load of bots.
Sheesh. I didn't expect this to wrankle with so many people.
The idea of a self-penalty is that we don't have to worry about robots.txt, or meta-tags, or anything else. The site is zapped and low volume and that's it: clean and easy for us, for good. If you can't understand why we prefer that, I can't understand why you can't understand!
And then there are the inevetable folks who see machiavellian intent somehow. It's OUR site for goodness sake.
The REASONS aren't what this thread was about though. The thread was about the MECHANICS of what we want to do. It seems that no-one wants to discuss that though. I guess some fell on stoney ground.
|The thread was about the MECHANICS of what we want to do. It seems that no-one wants to discuss that though. I guess some fell on stoney ground. |
Unfortunately the MECHANICS of what you want to do is something we would probably not want to share with the general public. There are of course your numerous threads at WebmasterWorld where penalties and bans are discussed. You could easily implement any number of techniques to get your site banned. From participating in bad link exchanges to using hidden text, it is all there for the taking.
What you won't find though are the MECHANICS for what you want to do. You'll need to play the role of mechanic and assemble your toolbox based on the various topics surrounding penalties and bans. And guess what, there are literally tens of thousands of them.
For many of us, it is difficult to look at a website from that perspective. If the site is toast now, then just let it go. Don't worry about it. Sounds like it was permanently toasted. Must have been something fairly severe to end up with that status.
Buy your own server and have your users dial up or remote access your files. Password protect your directories.
Have you even tried contacting google/msn/yahoo to see if they can permanently remove you from your index?
Having read both your post in december last year and yours now, I think I understand your situation. You speak the truth in that the site is still under your control now, but my strong impression that it won't be in the near future and you want it removed from the SERPs in such a way that it can't recover even if it turnes into other hands.
Unfortunately I have to disappoint you at this point. There were several posts here of people appearantly buying banned domains which they didn't know at forehand. When owner and content is totally new you can as the new owner request Google to lift the ban on that domain. So whatever your reasons are to permanently hurt the rankings of this site, after it is out of your control bans can be lifted.
OK, I give up. This particular site by the way will never leave our control. We just want to park it as it is, and then forget about it, with no maintenace needed and no new traffic. The ban would be the ideal solution for it for us, but I guess no-one will volunteer an easy mechanism. We'll have to experiement ourselves. No worries.