Is it time to de-index the Web?

[webcentric]

The Internet is an interconnected network of digital resources. It was conceived as such anyway. Along the way, someone decided it would be a good idea to index all the content on the Web to make it easier to find things and the race was on. The commercial prospects of such an endeavor were just too great to stop people from trying and try they have. Indexing the web is now HUGE business and everyone involved, from single-site webmasters and mom and pop storefront operations to amazon-sized companies and governments on every level all want their piece of the action. Everyone wants to be number one in Google and Bing and Yahoo and, blah, and blah and it doesn't seem to matter that that pursuit is ruining the Internet and shaping the world of information in a very scary way. It used to be that the Internet was a place where moving from place to place was the way to find things. Now, building natural connections between related information is seen as detrimental to the corporations who strive be the only authority on how people should find things on the Web. The result is that these mega-corps are dictating the terms of survival on the Internet when the only end game is truly their own survival. Jump through this hoop today (just to stay occupied) while the magician slices another rabbit's throat and tosses it in the pot.

Webmasters have provided the content that these major corporations are usurping for their own benefit. Without our content, they have nothing but a page of advertisements. Oh, wait a minute, they just built an advertising system to replace the actual Internet and they used our blood, sweat and tears to do it. Good one! Didn't see that one till it slapped me up side the head.

The Internet is on the verge of becoming a portal called Google and soon there will be no chance of promoting yourself for free because all the available promotion space on every page once used for the actual index will be bought and paid for. Indexed content is on the verge of becoming irrelevant and when it does, look out!

So, is it time to go back to networking the Web the way it was originally envisioned so people will actually be able to find things that matter to them in a natural way or are we all ready to concede that advertising is information and it's the only thing the word needs? If so, prepare for a new dark age because soon, there will be no medium left for free expression unless you're ready to chisel it out on a set of stone tablets a couple hundred thousand times and ship it around the world on the wings of a Dodo.

Is it now time to put the noindex value to it's proper purpose. I'm beginning to think noindex is the solution to a very big problem but it's gonna take a hell of a lot of noindexes to get the job done. What do these greedy [fill in the blanks] have without an index? Seems they may already have enough to abandon the pretense of providing search services at any time but I for one would be willing to put that assumption to the test. It's a simple question and a seemingly impossible proposition but I'll ask it anyway. Is it time to noindex the Web?

And I'll ask another question. How much of your bandwidth is Google sucking into it's carburetor to fuel the vehicle which is about to run you down in the street and or back up over you if it's already done so?

[webcentric]

@EditorialGuy
When does ROI enter into the picture when making the decision to allow Google to use/crawl your site? If they crawl your site 1000 times per day but send you a small fraction of that in real visitors, when does it become a losing proposition? Is it not logical that at some point it is just not cost effective to allow it? What other business would allow another business to use their product for free when there is no reciprocal gain in the relationship?

Yes, good point. I missed who actually posted this.

[aakk9999]

Why noindex? Why then just not forget about Google without going into extra steps to block it?

Yes, the bandwidth.... but unless you have a huge huge site, I do not think the bandwidth Google is using is an issue. I certainly do not think I am paying for more expensive hosting package just because of Googlebot crawling.

Anyway, noindex will not save the bandwidth - googlebot will still crawl.

Setting the "noindex" to the whole site seems like wanting to have some kind of revenge on Google. Otherwise, why bother with noindex?

Oposite of "love" is not "hate". Oposite of love is ignorance, e.g. "who cares..."

[taberstruths]

Disallow is a better option than noindex.

[webcentric]

Disallow is a better option than noindex.

If it's just about resources, that's true.

Setting the "noindex" to the whole site seems like wanting to have some kind of revenge on Google

Revenge is such a harsh word. I like to think in terms of "communication" on a level both parties can understand.


Then I was walking my dog and wondering to myself...

What if created a page, hmmmm, let's see...let's fill the top of the page with ads, OK, now, let's throw ads on the side, OK, now let's put a handful of links down at the bottom making sure visitors can't see them without scrolling. We'll make sure most of the links are somewhat related but there's no hard and fast rule on that either. Now let's put some tiny text at the top of the page to tell people what the page is about. How about

About 10 results (0.33 seconds)

Great, sounds like a very useful page unless of course you think it sounds like a link farm. It would be really great if I could now stop everyone else from using link farms so everyone has to use mine. What a concept!

Hypocrisy is an equally harsh term.

[RedBar]

I'm finding this thread very interesting since I'm going to have to make some serious recommendations to my fellow company directors.

We have been heavily Google slapped twice and it's all concerning our own images with January's image grab making us wonder sometimes just why we are now bothering with continually updating etc.

Since Google's grab we have generated in excess of one million 403 forbidden codes.

I'd be very interested in others' opinions as to the way they are thinking of going forward.

[LeeU]

Perhaps what is really needed is a search engine that indexes the web for links only, w/o regard to advertising, as webcentric stated, "the way it was originally envisioned". Granted, the organization that runs it would need some type of funding, but there might others, with the funds, who feel the same way but aren't sure it would work or is needed.

[piatkow]

I'm just contemplating what it would feel like if one day I politely said NO!

If everybody did they would simply start ignoring NOINDEX. Of course they would invent some new way of opting out to satisfy the lawyers but:
1. You would have to actively go to a Google/Bing/etc source to find out
2. You would need to update every page
3. Each search engine would probably require a different opt out method.

Looking at my site stats the use of direct links seems to have collapsed while encrypted search means that I can't see the keywords used on most searches.

[EditorialGuy]

You're implying that because you're finding success that your formula should be good enough for everyone else. Well it's not!

No, I'm not. I said "Go for it." But don't labor under the illusion that Google and other search engines will be brought to their knees if people who aren't getting any traffic from search engines tell search engines not to send them any traffic.

Also, are you 100% certain that the search engines will never send you enough traffic to justify the minuscule cost of serving pages to search crawlers? If not, is it really wise to take yourself off the search grid?

[lucy24]

Oh, all right, long version. Since search engines exist, humans use them as their primary means of finding previously unknown content. If the search engine doesn't have what you're looking for, the assumption is that it doesn't exist. Humans don't know from robots.txt, crawling and "noindex". They only know portal to internet.

Anyone seen the current Opera? If you don't spell a name exactly right, it defaults to google search. Not in a box conveniently adjacent to the address bar. In the address bar itself, where a traditional browser would say "Sorry, can't find webmasterworld.c". Obviously someone, somewhere, wants a browser to conflate search and navigation.

343, I wasn't referring to your grammar. There's a much deeper error.

[Marshall]

Is it time to de-index the Web?

Maybe more beneficial would be to clean the web. How many millions of useless, outdated pages/websites are floating out there that compete for space with active ones. As much a G likes to tout how all their algorithms penalize a rank low bad sites, they really have not taken a hard look at SERPs.

Marshall

[viral]

I think the point webcentric was trying to make and many of you are missing is knowledge graph/Google now/Carousels etc.

Ok at the moment it is only a small part of Google's index but eventually. Let us say 5-10 years down the track all queries will be returning knowledge graph type answers. Google admits this, they aren't trying to hide the fact. Want example of what this looks like search for "dementia symptoms" at Google.com. I hope the mods don't remove that keyword just this once. It doesn't really matter no one owns it except Google.

The above the fold experience is all knowledge graph and ads. One day and it's not far away Google will return results like that for every query! If you think they aren't then boy are you in for a rude shock! They will take it further though it will all of page 1 and then eventually it will be all of page 2 and 3 also.

This information they take from other sites and then they put ads around it. You will basically have to participate in adwords or get NO TRAFFIC! If you are the lucky site that they scrape to get the information you will then get an almost invisibly link to your site.

The problem with us webmasters is that we are to fractured. We are so busy competing against each other we can't see 500 pound gorilla in the room!

I personally think it is too late. Even if every webmaster here at WebmasterWorld blocked Google. The inertia Google has will carry them through this plan anyway. Plenty of other web sites out there who won't block Google in the hope of getting the scraps Google leaves under the table!

[webcentric]

Thanks viral for clarifying my position for me. Dead on. And yes, you're probably right, there probably is too much inertia now to stop what's happening. Doesn't stop me from speaking my mind and it's great to know there are a few others who are at least willing to acknowledge the fact that there's five-hundred pound gorilla loose in the room.

[webcentric]

Also, are you 100% certain that the search engines will never send you enough traffic to justify the minuscule cost of serving pages to search crawlers?

Death and taxes and change. Those I'm certain off. Everything else is speculation. This thread asks the question, what if? If I had the answer, I wouldn't have asked the question. I just pointed out a simple mechanism that's at everyone's fingertips here and dared to pose the question of what would happen if everyone used it all at once. It's called a philosophical debate or perhaps hypothetical thinking. BTW, it's the same kind of thinking that brought you the Internet in the first place and then the SE's we are discussing at this very moment. Again, I'm not certain of anything except that things are gonna change. Shaping the change, now that's an interesting proposition. If the folks at Google can do it, then so can I.

[hasek747]

Webcentric, no offense intended, but you sound like the guy who's girlfriend left him after a 5 year relationship and is now telling everyone that they should not get into relationships because they will end badly for them.

Yes, businesses sometimes fail. Same thing for a website and same thing for the grocery store around the corner.

If you do noindex, please let me know what your niche was so I can pick up the slack and milk Google for all its worth for as long as possible. :)

This thread looks more like a rant "masked" as a discussion to me.

[webcentric]

@hasek474 No offense taken. I expected to be mocked and demonized and ridiculed and marginalized when I started this discussion. No disappointment here. :)

[matrix_jan]

If only all my competitors added no-index to their websites... that would be something :)

Search engines "sucked" 200 GB from one of my websites this month so far. That's 10% of the overall traffic. If it was 200% of the overall traffic would I be less happy? Not a bit. My only goal is to be better than the competitors and adding a no-index is something that would put a smile on their face. And god I hate when they smile :)

Look at it this way, you have a shop on the street. It gets a lot of attention and you pay a tax just to be on the main street. If you move to the back of the building or to the basement you will not have to pay the extra tax and won't get all the attention. Now if you think your business will live without search engines and you won't feel the difference then what is keeping you from putting no-index?

And let's forget for once and all that all the webmasters will do certain actions together as whole. All webmasters will put no-index, all webmasters will not ban google and bing image hotlinking... this type of things are not going to happen. Let's get real. It's not even worth discussing.

Cheers

[viral]

If you do noindex, please let me know what your niche was so I can pick up the slack and milk Google for all its worth for as long as possible. :)

@hasek747 this kind of brovado is probably not called for here. Kind of reminds me of what happens on Warrior forum.

However I do agree yeah there is some ranting go on under all this and actually I am surprised the thread has lasted this long!

@matrix_jan I agree The webmasters getting together will never happen. We are all busy fighting over the scraps that Google is willing to leave us.

The thing is even Danny Sullivan believes that Google is moving away from being a search engine to a knowledge engine. When it completes that transition. It really won't need webmasters anymore. or do some out there still believe it will?

[ohno]

A knowledge engine that servs ads based on bid? Hahaha, that isn't knowledge, that is capatalism at it worst. So what if this thread is a rant? I think the people who have jumped through the Google hoops for a decade to be smacked in the face have a right to rant.

Marshall said it's time to clean the web, wasn't that what Google claimed to be doing? All they have done is taken away the good sites & filled the SERP's with big players & ads.
No index is 100% not the answer, the answer is to get away from Google & we all know that is easier said than done when they have such a massive market share.

Mark my words, when I'm long gone Google may not be around either, when you wipe out a lot of sites you put your eggs in one basket for the revenue you need. Large sites won't pay Adwords forever IMO if there is no need to.

Ironically they sent me an advertising pack last week with a sticker to put in my window saying "review us on Google"!

No thanks, better things to spend £££ on than line their pockets to get shafted down the line again.

[diddlydazz]

my 2 cents FWIW

noindex = no answer

as long as there are still 100 results for any specific query, the general G users would probably not even notice (that the rest of the web has suddenly disappeared)

the way G is heading is purely financially motivated, and why wouldn't it be? they are not the fluffy friend we all promoted like crazy when they first come on the scene

they have slowly started taking out every niche, starting with the most commercial, hotels, flights et al

they do exactly what they preach to us NOT to do, so it is understandable why there is some ranting going on.

for those who have not been affected and refuse to see the direction G is heading, then maybe they can report back after G starts encroaching in their verticals.

and the knowledge graph is very alarming, this will be a serious game-changer and will affect untold niches, especially those "less commercial" information-only websites that made the Internet what it is today.

so yes, the future is scary if you intend to rely on google for organic traffic and i truly believe that G will have a negative effect on the future of the Internet as an open freely-available source of information.

the commercial verticals going pure pay-to-play, that is inevitable

but when G is number one for terms like "earth"

and a typical acronym search brings back a top heavy google "answer", "knowledge" call it what you will (which incidentally was scraped from some source or another), then for me the value of the Internet will slowly demise

but, IMHO, noindex is not the answer

and i am not completely sure there is an answer when G has the branding and market share that it enjoys, which ironically, is mainly down to the people who they are slowly putting out of business promoting them heavily in the early days.

[austtr]

..... that Google is moving away from being a search engine to a knowledge engine. When it completes that transition. It really won't need webmasters anymore. or do some out there still believe it will?

Of course they need webmasters... where else are they going to get free content that can be folded into screen real estate that carries their products and services.

Google can deliver information via whatever format it wants (SERP's, knowledge graph whatever).... nothing changes the fact that they don't own the information. We do, our websites and the intellectual property tied up in the sites is NOT Google's to use as they please. But that is just what they are doing.

When did Google acquire the right to take information/answers/resources etc that we created/published/made available and use that for their own commercial benefit. I must have missed that email.

A scorched earth de-index approach will have no effect... 10, 100, 100,000 of you and Google would not even notice. But start getting influential media to ask questions that shine a light on their business practices, and get some class actions going that try to force Google to pay for the content it uses.... then maybe you might have an effect. But you need intervention at government level for anything close to that to happen.

And the cow jumped over the moon....

[netmeg]

The ones who will make it are the ones who have already figured out how to keep Google in its place, by developing other channels. Yea yea yea, market share, niche, blah blah blah, Google's the only source of real traffic, blah blah blah. Bull.

If you're getting decent organic traffic now and you haven't figured out to leverage that to support and/or finance your other channels, then you're in more trouble than the guy with no organic traffic at all.

[ohno]

Please enlighten us netmeg how to grow WITHOUT Google. If it is that easy why do all the huge corporations still PAY to appear top of their SERP's? You say bull I say show me otherwise.

[zeus]

I have a few sites, where I have a no index, but for google only. The visits to those sites come from other SE and social Networks. I did it from the beginning of those sites, so I will not be dependent on Google. Also a new site which is almost done, I have the option for users to block google, so there content dont show there.

[netmeg]

Ohno, I've been showing people otherwise (and doing it) for years. If you missed it, that's on you.

[mrengine]

If it is that easy why do all the huge corporations still PAY to appear top of their SERP's? You say bull I say show me otherwise.

Join Facebook, G+, etc., create content on THEIR sites, and watch the traffic start flowing. It's is simple as that. But in the fine print I forgot to mention it may take a few years for that to happen unless you post something that goes viral.

Our ecommerce site does six figures each year, with product that we produce. We are active on Facebook, Twitter and G+. Here's a breakdown of the YTD sales originating from each one...

Facebook = 0 sales
Twitter = 0 sales
G+ = 0 sales

Maybe these social sites are good for those that have the information sites to make a few dollars from tire kickers that will click adsense ads, but for many businesses I think it's a complete waste of time. I'd get a better return on my time if I went door to door with a cold sales pitch!

Here's the thing. Our primary site is not dependent on Google at all. Maybe being in business longer than Google has existed has something to do with it. But Google screwed up our ranks too, and I'm fine with that because we don't need them at all for this site. But if you have not been in business for many years, with a reputation in your niche, you are flat out screwed. To be found, new entrants need to join Google's pay to play network or invest years in building recognition elsewhere. And all that "content is king" garbage that people talk about, is pure rubbish that belongs in the dump. Google's rising stock value and profits are the laws by which their algorithm operates.

Bing respects our reputation in organic search, and that is why I spend money with them. Google, on the other hand, is in love with Amazon so much that they get 3, 4 and even 5 listings in a row on the front page of the search results! Because of this, I will not authorize Google to get even the sweat off my n**s on a hot summer's day.

Plain fools, shills or whatever you want to call them are the ones that keep preaching the same Google gab rubbish that may have been true 5 years ago. Wake up people, GOOG is a public company with shareholders that want more and more earnings each month. Many of GOOGs shareholders are right here in the forums too, which makes me think that many people talking about how good Google is actually have hidden agendas.

[hasek747]

@viral

That's not 'bravado' at all, I'm quite serious. If such a movement would go into effect (where people would start noindexing their sites) I would work really hard to figure out a way to find those people so that I can build websites based on what they have noindexed. Kind of like a new way for doing keyword research.

[netmeg]

Here's the other thing to remember - if it weren't Google, it would be Microsoft, or Comcast or Verizon or AT&T or Amazon or somebody else. Google just got to this particular place first, and the others would love to step in wherever they find a chink in the armor.

So when you're thinking about reinventing the web for the future, it's not going to be enough to make it Google proof.

[ohno]

Ohno, I've been showing people otherwise (and doing it) for years. If you missed it, that's on you.

Well I guess a fair few others have missed that golden info too as some seriously big players in our field continue to spend with Google. If no one needed them they wouldn't waste money on them.

[webcentric]

It's almost Thanksgiving in America. I'm thankful for the fact that 10 years from now, organic search will be ancient history and this particular forum will have no purpose except to document the smoke and mirrors our magician used to make the golden goose disappear right before our very eyes.

There will be other magic shows of course, each as pleasantly intoxicating as the last. The show must go on. This much is certain. Enjoy the show! Live for the moment! Carpe diem! Rest in peace!

[edited by: webcentric at 3:22 pm (utc) on Nov 26, 2013]

[ohno]

Here's the other thing to remember - if it weren't Google, it would be Microsoft, or Comcast or Verizon or AT&T or Amazon or somebody else. Google just got to this particular place first, and the others would love to step in wherever they find a chink in the armor.

So when you're thinking about reinventing the web for the future, it's not going to be enough to make it Google proof.

Would you like an auto industry where you could only buy Ford cars? Or TV's that only Sony sell? Lack of competition is bad for everyone.