Is it time to de-index the Web?

[webcentric]

The Internet is an interconnected network of digital resources. It was conceived as such anyway. Along the way, someone decided it would be a good idea to index all the content on the Web to make it easier to find things and the race was on. The commercial prospects of such an endeavor were just too great to stop people from trying and try they have. Indexing the web is now HUGE business and everyone involved, from single-site webmasters and mom and pop storefront operations to amazon-sized companies and governments on every level all want their piece of the action. Everyone wants to be number one in Google and Bing and Yahoo and, blah, and blah and it doesn't seem to matter that that pursuit is ruining the Internet and shaping the world of information in a very scary way. It used to be that the Internet was a place where moving from place to place was the way to find things. Now, building natural connections between related information is seen as detrimental to the corporations who strive be the only authority on how people should find things on the Web. The result is that these mega-corps are dictating the terms of survival on the Internet when the only end game is truly their own survival. Jump through this hoop today (just to stay occupied) while the magician slices another rabbit's throat and tosses it in the pot.

Webmasters have provided the content that these major corporations are usurping for their own benefit. Without our content, they have nothing but a page of advertisements. Oh, wait a minute, they just built an advertising system to replace the actual Internet and they used our blood, sweat and tears to do it. Good one! Didn't see that one till it slapped me up side the head.

The Internet is on the verge of becoming a portal called Google and soon there will be no chance of promoting yourself for free because all the available promotion space on every page once used for the actual index will be bought and paid for. Indexed content is on the verge of becoming irrelevant and when it does, look out!

So, is it time to go back to networking the Web the way it was originally envisioned so people will actually be able to find things that matter to them in a natural way or are we all ready to concede that advertising is information and it's the only thing the word needs? If so, prepare for a new dark age because soon, there will be no medium left for free expression unless you're ready to chisel it out on a set of stone tablets a couple hundred thousand times and ship it around the world on the wings of a Dodo.

Is it now time to put the noindex value to it's proper purpose. I'm beginning to think noindex is the solution to a very big problem but it's gonna take a hell of a lot of noindexes to get the job done. What do these greedy [fill in the blanks] have without an index? Seems they may already have enough to abandon the pretense of providing search services at any time but I for one would be willing to put that assumption to the test. It's a simple question and a seemingly impossible proposition but I'll ask it anyway. Is it time to noindex the Web?

And I'll ask another question. How much of your bandwidth is Google sucking into it's carburetor to fuel the vehicle which is about to run you down in the street and or back up over you if it's already done so?

[taberstruths]

I am about 6 months away from deciding to disallow googlebot. At this point the cost of the investment (bandwidth used by googlebot) is less than the return (visitors provided by google.)

[EditorialGuy]

Dunno about you, but I get thousands of referrals each day from the "portal called Google" and other search engines. Those referrals don't just benefit me; they also benefit users who are looking for the information I provide.

Becoming a hermit and a hoarder wouldn't help me or my readers.

[taberstruths]

EG, Glad you get those visitors. I no longer do and I haven't changed the content. I increased site speed to a score of 91 desktop and 85 mobile, increased user interaction to time on site greater than my competition and bounce rate lower than my competition, and all I have seen is a decrease in impressions from search. It is a responsive design so it is mobile friendly, and I have outstanding social signals. Every time a positive change was made, a drop in referrals corresponded. So honestly, since my site does fine with Bing/Yahoo traffic, pinterest traffic, facebook traffic, and and even G+ traffic, and I get no GLove, why feed it?

[webcentric]

Dunno about you, but I get thousands of referrals each day from the "portal called Google" and other search engines.

"...for all they that take the sword shall perish with the sword"

Matthew 26:52

I found that using Google which decided for some reason that a Wiki is somehow more authoritative than the actual source of the information.

Gospel according to Google, now that's a concept for you.

[netmeg]

Why would you expect online to function differently than offline?

[ohno]

Well I haven't made a penny for three days now. Prior to that I was having the best week in months. Maybe everyone is watching TV, maybe the economy is broken, maybe no one wants my wears any more. Or maybe, just maybe Google is screwing me over yet again.

Google are so far up their own backside they don't care what happens. Anyone who thinks they are interested in anything other than money is deluded. I've got to hand it to them, they are VERY clever. It took years to get to where they are, give products away for "free" to mine data. They know more about me that I do! Time to face the sorry facts, they can (& do)do what the hell they like now, soon the Internet will follow the highstreet. All those nice little independant shops that you used to see but never supported will soon be gone.

I never wanted to be rich, I'm quite happy working hard & earning a half decent living.

It's impossible with new Google.

Oh, & if you think sticking the money into Adwords will help do yourself a big favour, stick that money on the horses, you'll have a bigger chance of winning & won't have a big Google hand up your ass!

So sad that people have actually stopped linking to sites because they are scared these clowns in Mountain View will slap their site for doing so.

[edited by: brotherhood_of_LAN at 4:08 pm (utc) on Nov 25, 2013]
[edit reason] fixed typo [/edit]

[mrengine]

Is it now time to put the noindex value to it's proper purpose. I'm beginning to think noindex is the solution to a very big problem but it's gonna take a hell of a lot of noindexes to get the job done.

Google has definitely become a giant leech and is sucking the life out of many small companies. Using noindex to stop Google from monetizing your content may only bring personal satisfaction. But if such a movement were to gain momentum, it might help to give other search engines an advantage.

[brotherhood of LAN]

While it's an interesting concept moving away from search engines, IMO they are likely to be adding to the overall productivity of the web. It would be harder to find things if they weren't there. If that is true, we would need a replacement.

[webcentric]

Why would you expect online to function differently than offline?

I don't. I understand that business is war. I've read Miyamoto Musashi. I get it. Also read Darwin.

Not sure if that answers your question or not. It was a bit vague.

[ohno]

Using noindex to stop Google from monetizing your content may only bring personal satisfaction. But if such a movement were to gain momentum, it might help to give other search engines an advantage.

True, but I fear the uptake would be few & far between & others have a LOT of catching up to make on Google. Bing could make soem big freinds right now if they reversed some of what Google has done since May Day. Where's Bing shopping in the UK? Where's the one box results you used to see on Google on Bing? Where's Bing Checkout now Google Checkout has gone? Loads of people hate Pay Pal which is why many took on Google Checkout. Where's Bing Chat, Google Chat made it easy to add live chat to your site.

[webcentric]

Yup, replacement would be a huge challenge but the replacement may have been there all along, it's called linking. Who said a page of links was a bad thing? Answer. Google. Who said blog comments and links in signatures are a bad thing. Google. BTW, I'm just using Google to lump all entities that uses similar means of coercion into one rotten apple barrel. Google has created a version of licensing for the Internet e.g. a license is the right or privilege (acquired by paying a fee usually) to do something that would otherwise be illegal. Google's form of illegality is to do something on your own that would keep it from succeeding in it's agenda.

[ohno]

Yep, why were they so keen to kill linking? Because links cut THEM out of the equation! No need to visit them with good old links.....

[brotherhood of LAN]

Links are good, and I'm happy for sites to freely link (though they should take care in maintaining the links).

But they're not a replacement for a search engine. Users would have to know which site to find the link on. You could call it an 'authority' but you arrive back at the same problems of subjectiveness that the site owner has, in comparison to some algorithms, it'll always be flawed from another person's POV.

[webcentric]

Here's some Kool-aid for everyone. Virtually, every discussion on the subject of noindex I've encountered in my career has been focused on selectively picking things to keep out of the index because you might not want to draw attention to them or because you don't want them diluting your page rank or whatever which just glosses over the fact that its actually a very powerful tool for removing huge chunks of content from the indexed web with a few keystrokes (theoretically anyway). Again, I ask, where would the power brokers be if everyone did that? Some of the most controlling governments in history have learned the hard way just how much power the mob possesses. I'll just reference the French and American revolutions for a couple of examples. Common ground is a very powerful place to be and the more people that stand on it (or are forced to stand on it), the more powerful it becomes. Search may be useful but its coming at a steep price to your neighbors even if you aren't feeling the pinch.

<snip>

[edited by: aakk9999 at 5:09 pm (utc) on Nov 25, 2013]
[edit reason] Please refrain from a call to action. We are discussing, not starting a revolution. [/edit]

[webcentric]

Links are good, and I'm happy for sites to freely link (though they should take care in maintaining the links).

But they're not a replacement for a search engine. Users would have to know which site to find the link on. You could call it an 'authority' but you arrive back at the same problems of subjectiveness that the site owner has, in comparison to some algorithms, it'll always be flawed from another person's POV.

You're certainly right that linking isn't a replacement for search. But that's a different discussion than showing the big boys that they have gone too far and that they still don't hold all the cards. I'm just pointing out that we've given them their power and that the power they've been given isn't as everlasting as they might think it is.

[SevenCubed]

Is it time to de-index the Web?

Your timing for a thread like this is impeccable. As a matter of fact, on my personal business domain, I changed my robots meta tag to noindex, nofollow, noarchive about a week ago. I'm closing shop at the end of the year and just wanted to get a head start on getting my pages de-indexed.

I may still stick around to develop a personal project that I haven't had time for. But even that, I'm now seriously considering putting it into book form rather than internet format. I wanted the information to be freely given, not someone else to scrape it and profit from it. Now instead it looks like I'll have to go the traditional route and print a commercial book instead. I guess I can just pass on the bulk of the profit to small community organizations who are trying to make a difference. We'll see.

The winds of change are blowing for me again. I've NEVER been one to sit around and wait for things to improve. I create my own change, dance to my own drumbeat <snip>. IMHO the internet right now is lost for the foreseeable future.

I had hopes of moving as much commerce to the internet as possible. We could have built large global fulfilment centres in desert land areas and reclaimed green space to keep the planet alive and vibrant for our descendants.

[edited by: aakk9999 at 5:12 pm (utc) on Nov 25, 2013]
[edit reason] Removed link to off-topic youtube video [/edit]

[webcentric]

@SevenCubed I hear you loud and clear. As for this.

I've NEVER been one to sit around and wait for things to improve.

Me either. We should build our own Internet. You're welcome on my wavelength anytime. :)

[edited by: aakk9999 at 5:26 pm (utc) on Nov 25, 2013]
[edit reason] Forum Charter [/edit]

[lucy24]

We could have built large global fulfilment centres in desert land areas and reclaimed green space to keep the planet alive and vibrant for our descendants.

Well, thank Heaven it didn't happen, then, because the component parts of this sentence don't interlock in any way; in fact some elements are mutually exclusive.

Search engines exist.

[brotherhood of LAN]

I must have misunderstood webcentric,

Are you against search engines in general, or just the way that the current prominent search engines conduct their business? The post title implies it's the former.

[diberry]

It may be lost for now, but young friends keep telling me they only use search for targeted hunting. They "discover" cool sites and products/concepts they "didn't know they wanted" through social media.

And apps. Don't forget the apps - we have yet to see all the ways they're going to change user behavior.

I think there will always be a use for search engines as curators of targeted info, but that's not all the web is for. It's called "surfing" for a reason, and links are the foundation of that surfing activity.

Before I deindex anything, I'm just going to work on linking strictly as I believe serves my visitors. In marketing, you consider your audience primarily and worry about what "method of transportation" they'll use to get to your "store" secondarily. Because if they're motivated enough to get to you, they will. Being #1 in Google is a good way to be seen, but it's better if you can motivate the audience to seek you out.

[diberry]

We could have built large global fulfilment centres in desert land areas and in reclaimed green space to keep the planet alive and vibrant for our descendants.

I read it this way (with the bold "in"), Lucy, in which case I think it does all hang together unless I'm missing something.

[SevenCubed]

We could have built large global fulfilment centres in desert land areas and reclaimed green space to keep the planet alive and vibrant for our descendants.

Yeah, my bad :) I just reread it and realize I should have said:

We could have built large global fulfilment centres in desert land areas while reclaiming green space to keep the planet alive and vibrant for our descendants. That's what I was really trying to say. Obviously I need more coffee.

[taberstruths]

Well I just took the first step. I just disallowed all my images to be indexed. The quality of traffic and the purpose of the traffic was such that it honestly was a no brainer. Who needs people coming to just bounce or steal your pics?

[webcentric]

Are you against search engines in general, or just the way that the current prominent search engines conduct their business? The post title implies it's the former.

Indexing the web and making it searchable is useful. Using the pretext of creating a useful tool to scrape the entire web, suck people into relying on your index and then basically replacing the index with ads is going too far. I'm simply suggesting that they've monetized everything that wasn't theirs to start with and now they're turning the work of millions of independent people into something that they want to call theirs. The simple fact is that if the world flagged it's content as not index-able for one month and Google actually obeyed the instruction as is their responsibility to do so, I'm fairly certain it would generate a few serious discussions in the board room. Yes a lot of people would probably be committing an act of self-destruction to accomplish such an impossible feat and for that reason alone it will probably never happen. But the fact remains, oppression taken too far is subject to backlash. I can imagine someone at Google right now headed over to w3C to see if they can't get the noindex attribute value removed from the next specification (just in case the population decides it's had enough). So, yes, I'm saying Google is using your labors against you under the pretext of some temporary benefit.

He who would trade liberty for some temporary security, deserves neither liberty nor security.

Not my words, (see Benjamin Franklin)

[EditorialGuy]

OK, go for it. "Noindex" everything. Let us know how you like being part of the private Web.

[webcentric]

Search engines exist.

Yes and their little robots come around to my site every day and ask if they can use my content for their own benefit with no promise of anything in return. I'm just contemplating what it would feel like if one day I politely said NO! I don't suggest anyone else should think like this because it's a seditious and dangerous way to think. In fact, forget I mentioned it. I didn't realize I was muttering in such a public place.

[steve40]

@ webcentric

I am not sure I would go that far but I for one decided 3 months ago to remove 15,000 images from Google's Index due to the bandwidth they used and very little in visitors , the Image side of Google is designed from the ground up to keep visitors on Google and not send them to the people who created the images

I suspect many many webmasters have looked at bandwidth server speed etc. etc. and decided images are a liability not an asset

Steve

[taberstruths]

@EditorialGuy
When does ROI enter into the picture when making the decision to allow Google to use/crawl your site? If they crawl your site 1000 times per day but send you a small fraction of that in real visitors, when does it become a losing proposition? Is it not logical that at some point it is just not cost effective to allow it? What other business would allow another business to use their product for free when there is no reciprocal gain in the relationship?

[webcentric]

OK, go for it. "Noindex" everything. Let us know how you like being part of the private Web.

If indexed site's are getting pushed off the grid anyway, what's the difference?

If you want to have a discussion fine. If you want to churn out FUD in support of some implied conclusions you want everyone to believe is fact, go for it. You're implying that because you're finding success that your formula should be good enough for everyone else. Well it's not! Just because you haven't been crushed into oblivion yet doesn't make you right. Ask the person who has "Please index my content" plastered all over their site but can't get a decent or consistent flow of traffic from an SE how valuable content="index" is to them.