Is it time to de-index the Web?

[webcentric]

The Internet is an interconnected network of digital resources. It was conceived as such anyway. Along the way, someone decided it would be a good idea to index all the content on the Web to make it easier to find things and the race was on. The commercial prospects of such an endeavor were just too great to stop people from trying and try they have. Indexing the web is now HUGE business and everyone involved, from single-site webmasters and mom and pop storefront operations to amazon-sized companies and governments on every level all want their piece of the action. Everyone wants to be number one in Google and Bing and Yahoo and, blah, and blah and it doesn't seem to matter that that pursuit is ruining the Internet and shaping the world of information in a very scary way. It used to be that the Internet was a place where moving from place to place was the way to find things. Now, building natural connections between related information is seen as detrimental to the corporations who strive be the only authority on how people should find things on the Web. The result is that these mega-corps are dictating the terms of survival on the Internet when the only end game is truly their own survival. Jump through this hoop today (just to stay occupied) while the magician slices another rabbit's throat and tosses it in the pot.

Webmasters have provided the content that these major corporations are usurping for their own benefit. Without our content, they have nothing but a page of advertisements. Oh, wait a minute, they just built an advertising system to replace the actual Internet and they used our blood, sweat and tears to do it. Good one! Didn't see that one till it slapped me up side the head.

The Internet is on the verge of becoming a portal called Google and soon there will be no chance of promoting yourself for free because all the available promotion space on every page once used for the actual index will be bought and paid for. Indexed content is on the verge of becoming irrelevant and when it does, look out!

So, is it time to go back to networking the Web the way it was originally envisioned so people will actually be able to find things that matter to them in a natural way or are we all ready to concede that advertising is information and it's the only thing the word needs? If so, prepare for a new dark age because soon, there will be no medium left for free expression unless you're ready to chisel it out on a set of stone tablets a couple hundred thousand times and ship it around the world on the wings of a Dodo.

Is it now time to put the noindex value to it's proper purpose. I'm beginning to think noindex is the solution to a very big problem but it's gonna take a hell of a lot of noindexes to get the job done. What do these greedy [fill in the blanks] have without an index? Seems they may already have enough to abandon the pretense of providing search services at any time but I for one would be willing to put that assumption to the test. It's a simple question and a seemingly impossible proposition but I'll ask it anyway. Is it time to noindex the Web?

And I'll ask another question. How much of your bandwidth is Google sucking into it's carburetor to fuel the vehicle which is about to run you down in the street and or back up over you if it's already done so?

[n0tSEO]

Why noindex? Why then just not forget about Google without going into extra steps to block it?

I know a SEO consultant who has banned Google completely from his website. I liked that move. :) SMART.

But like aakk9999, I wouldn't do that. Ignoring is enough. My life doesn't rely on Google, nor my business does (Google traffic to my websites is really small compared to other sources).

webcentric, I understand your frustration. But if we stop and consider the whole situation, it becomes clear that it doesn't have to be this way--- we don't have to follow Google's way of (business) life.

Google is only as powerful as we make it.

Like fears; they grow bigger as we surrender to them.

Since search engines exist, humans use them as their primary means of finding previously unknown content. If the search engine doesn't have what you're looking for, the assumption is that it doesn't exist.

Not quite, lucy24. If the search engine (you're using) doesn't have what you're looking for, try another search engine. And another. And another. Then try web directories, webrings (oh yes, they still exist!) and social networks.

I got my first Internet connection in 2001 and I've never stopped to Virgilio (an Italian search engine) set as my default search on my old IE browser. No. I was a 16 year old newbie web user but I already knew that if Virgilio didn't have something, maybe Altavista had it, or maybe Ask Jeeves or Lycos. I left no resources untapped.

Google sometimes seems to believe that the average web user is dumb. Well, s/he isn't. People might get lazy, but are not dumb. They don't need spoon-fed results; they need to be given the tools to get the results they're looking for.

It's like entering a library. You have all the tools and the staff available to help you with your search, but YOU take responsibility for your search, not the library.

noindex = no answer

Yeah, not on search engines. But you can still get found via other channels, including web directories.

I expected to be mocked and demonized and ridiculed and marginalized when I started this discussion. No disappointment here.

You're kidding me, aren't you? :) This is the most insightful discussion I took part into in a looong time. No, really-- THANK YOU for it!

Well, I'll conclude my thoughts (for now) with something I consider a simple truth--

It's NOT search engines (as tools) to be the issue. It's some search engine business' god-like ambitions to threaten the Web. At least, in appearance.

Until we start laughing in the face of fears. ;)

[lucy24]

If the search engine (you're using) doesn't have what you're looking for, try another search engine. And another. And another. Then try web directories, webrings (oh yes, they still exist!) and social networks.

You're speaking as a webmaster, which by definition means someone who knows stuff the ordinary human doesn't.

[webcentric]

Um, all that means is don't list the DMOZ title. Doesn't have anything to do with bots.

My point was about what isn't there e.g. index or noindex. I should have been clearer.

I understand your frustration.

Frustration is a tool. Insights come and go but it's wonderful to have one occasionally, even if it takes blowing off a little steam to get it to rise to the surface of the palaver. Ranting for it's own sake is pointless from my perspective except to temporarily release stress (if that works for you). Inciting a conversation, by whatever means necessary, is a fruitful exercise in my book. I'm here to learn and grow and evolve and learning starts with turning your known world on its head every once and awhile. Or at least that's one way to get a new perspective. I'm delighted with the results.

[webcentric]

You're speaking as a webmaster, which by definition means someone who knows stuff the ordinary human doesn't.

Now you're referring to another related concept, e.g. the dumbing-down of the Internet. Reduce everything to an advertisement and limit choice because people are lazy and stupid. Used to be you needed a bit of intelligence to use this medium. Now any idiot on the freeway can do it with a cellphone in one hand and a cup of coffee in the other.

[ohno]

Nonsense. Google won, humans failed. People aren't googling for amazon/wikipedia/ebay because they don't know the URL. They're googling because they don't know that there is an URL. There is absolutely nothing amazon/wikipedia/ebay/your-own-site could have done to prevent this behavior. If you want to point fingers, blame the browser designers who have deliberately blurred the distinction between navigation and search.

That is an excellent point, can anyone remember which browser started this trend?

Humans were trained, and google trained them, Amazon should have trained them.

How did Google train them though? By my memory this behavior started when the browsers URL bar became a search bar too?

I was a 16 year old newbie web user but I already knew that if Virgilio didn't have something, maybe Altavista had it, or maybe Ask Jeeves or Lycos. I left no resources untapped.

An excellent point, but how many people do you now know who do this? If I ask everyone I know none know this, they think the Internet IS Google. Even I am guuilty of this, using the major tool for what, 14 years means it's automatic to search via them. Only in the past 12 months have I started to use Bing as I just can't find what I could before on Google. Bing is FAR from perfect but when Google forces the same sites down my throat I look elsewhere. The majority, at the moment, don't IMO.

[webcentric]

If I ask everyone I know none know this, they think the Internet IS Google. Even I am guuilty of this, using the major tool for what, 14 years means it's automatic to search via them. Only in the past 12 months have I started to use Bing as I just can't find what I could before on Google. Bing is FAR from perfect but when Google forces the same sites down my throat I look elsewhere. The majority, at the moment, don't IMO.

And don't forget the very many SEO operations out there (and in here) that have made millions pushing Google on the world and are now scrambling for a new way to justify their existence before they've completely lost a handle on the game. I stopped using Google Search except in dire situations (or to simply follow along with the changes which are hot topics in these boards these days) a few months ago and pretty much everyone in my household has done the same without so much as a whimper. Love the idea of MS's new Scroogled line of attire too. Some may not want the world to tear apart it's precious Google but the competition will keep working at it, webmasters will continue to resist and who knows, perhaps even a government or two will get in on the action. Who knows if a class action suit against Google will be the subject of a historical Wikipedia article someday. I'd probably read it. "May you live in interesting times."

[webcentric]

And since I'm pretty much up late by myself here, let's take another tack on the subject. How many of you here are jumping on the structured data bandwagon these days? Are you even thinking about the consequences of the notion that information, all information should be machine readable? Do you see any connection to knowledge graphs? Now you can work directly for Google as a volunteer, structuring your data to so it can be displayed on Google directly. Why host information on my site if people can just read it on Google and go on their merry way? A good follow up to this topic might be, is machine readability just another mechanism for enforcing conformity on the unruly Internet populace?

[viral]

There is another point we need to consider and that is that us SEO types are Google's biggest competitors. Forget about Binghoo and the other searchduds.

SEO's take more food out of the mouths of Googlers than all of Google's other competitors put together.

We compete with adwords and in that respect we so far have done a pretty good job. So SEO's who say that we need to work with Google are just advocating a path of diminished returns.

Google see's us as the evil competition. Those who don't believe that are fooling themselves. Google is a listed company, it is the job of Google's leaders to see us in that light. If they didn't they wouldn't be doing their jobs properly.

All the other malarkey about SEO's being necessary that is spouted by Matt Cutts et al is just a case of keeping your enemies closer.

I just wish that webmasters and SEO's looked at Google the same way Google looks at us. As the enemy and something to be fought.

[n0tSEO]

If I ask everyone I know none know this, they think the Internet IS Google.

That's the sad thing with the majority (thankfully, not all) of the average web users. I don't think this is a problem that can be solved overnight, because it requires education, and education needs a plan, programs, slowly raising awareness that Google and Facebook are not THE Web, but a part of it.

Ideally, if a person picks Google for their default search engine, it should be a choice, not the only option available.

You're speaking as a webmaster, which by definition means someone who knows stuff the ordinary human doesn't.

At 16 I just barely knew how to use a computer to use the Internet and draw pictures in Paint. I didn't even know what a 'webmaster' was or how to build a website.

It was curiosity to lead me to not stop to one search engine. I wanted to know if there was more out there. :)

I'm here to learn and grow and evolve and learning starts with turning your known world on its head every once and awhile.

Me too, webcentric, that's why I'm thankful for this thread. I've had this kind of discussion offline for months but still no chance online, unless we mention the random blog comment.

We use services that force you to have a Google account, or a link to Google Analytics, or Google+ if you contribute to a big blog and that's a requirement to write for them. There are constraints that won't let us abandon Google right away even if really we want to. So raising awareness is the way to go if we want to change something.

P.S. SEO = Google? Uhm--- No. Even though that's what the majority thinks.

[webcentric]

SEO types are Google's biggest competitors

I've come in some way to understand this point. You could say that about both white hat and black hat SEO. It's easy and inviting for an entity in a position of power to declare something illegal to justify it's own ends. Notice how the list of transgressions keeps growing and the number of hoops you need to jump through to get your biscuit are also increasing. It's a game of diminishing returns.

[lucy24]

How many of you here are jumping on the structured data bandwagon these days? Are you even thinking about the consequences of the notion that information, all information should be machine readable?

Funny you should say this, because just a few days ago I added a few lines of microdata to one page of my art studio's site. Never done it before and don't anticipate ever doing it on a wider scale, because I see only one legitimate use: when the www page itself is secondary to something that exists in real life. I don't particularly care if people come to the studio's website or not, so long as they come to the listed event.

But it frankly gives me the fantods to have my name attached to a page that doesn't validate. (Seven microdata tags or attributes = 7 validation errors. You better believe I checked ;))

[ohno]

Re:structured data, looked at it & decided against it. Why would I want to give them even more info for them to monitize down the line? Will structured data become the new keywords for PPC?

[webcentric]

I don't particularly care if people come to the studio's website or not, so long as they come to the listed event.

No doubt, there are biscuits to be had for jumping through the hoop. And I do get you where validation errors are concerned. Visual Studio Intellisense hates structured data tags and I hate seeing my code highlighted in error red.

[EditorialGuy]

How many of you here are jumping on the structured data bandwagon these days?

I began adding schema.org article markup and link rel= pagination to our site's evergreen articles last August, and I've seen significant (in some cases, huge) jumps in traffic for many of our multi-page articles. Also, if schema markup can help the search engines better understand the nature of our pages and their content (e.g., "name," "description," and "articleBody"), it makes sense to use it.

For listings-type content (business addresses and hours, for example), whether structured markup is a good thing probably depends on what kind of business you're running. An emergency veterinary hospital, a shoe-repair shop, or the local McDonald's franchise probably is less interested in getting traffic on its Web site than in making it easier for prospective customers to phone or visit. A Yellow Pages directory, on the other hand, is going to want searchers to visit its Web site because the Web site *is* its business.

When deciding whether to implement structured markup, it's better to be logical than to get caught up in rhetoric like "If I use structured markup, I'm just helping the rich, evil people at Google get richer."

[ohno]

Lol, yep,sell your soul a little bit more. If you think it's to make their life easier to send you free traffic forever dream on. Will people never learn? I spent a decade jumping through Google hoops & to be frank I no longer trust them to tell them anything they probably don't already know about my business. Come back in 5 years time & tell us how you are doing when they have probably decided the data you GAVE them is now going to cost YOU.

[Rosalind]

Well I just took the first step. I just disallowed all my images to be indexed.

I did this on my most image-rich website, and found no downside.

As for the schema.org markup, I generally avoid using it, except for author attribution. But I agree with EditorialGuy, it's a decision that depends on the type and purpose of the website.

[mrengine]

Re:structured data, looked at it & decided against it. Why would I want to give them even more info for them to monitize down the line? Will structured data become the new keywords for PPC?

I agree that using structured data has potential risks in the future. Just like with what Google did with image search, by showing full size images now, it's very clear Google wants to keep people on their properties and not send people to anyone elses website. Structured data is a nice concept, but Google is sure to abuse that in the future too. Giving Google the ability to interpret data with structured data will allow them to answer keyword searches on a granular level. If we look to how Google handled image search, history will likely repeat itself. What was originally seen as helpful for webmasters will see the life sucked out of it by Google's greed.

As far as I am concerned Google will use structured data as a trojan horse in the future, much in the same way Google uses Analytics to evaluate website traffic on a granular level across many, many websites.

[webcentric]

Also, if schema markup can help the search engines better understand the nature of our pages and their content (e.g., "name," "description," and "articleBody"), it makes sense to use it.

"Understand" or "Use?"

Yes, there are treats to be had for helping to train the bot, for conforming yourself to it's nature. First you simply need to ask to be used e.g. "index" me. Then you need to morph yourself into digestible bot-food.

depends on what kind of business you're running

And you're right, for some, the downside is already plainly obvious. It is purely self-defense to not use structured data if the data is your business. But what if editorial content benefiting from a boost in listings today, is actually in jeopardy and you just don't see it yet? What's next? Perhaps a knowledge graph or some mechanism will start showing your lead content, like the home page of a blog. Hmmm. A Google magazine, Google News, etc. That would probably still be good for you if the "Read More" link actually goes to your site. But what happens on the day when someone can read your whole article, surrounded by its own ads instead of yours and the visitor no longer has any incentive to visit your site. Sound absurd? Perhaps? But it's exactly what's happening to some not so fortunate as you already. I would suggest imagining that the slippery slop really is slippery and keep in mind the distance between science fiction and science fact (or even the distance between benevolence and oppression) have proven to be just a matter of time.

This post is bound to raise the argument that Copyright protections would prevent the wholesale republication of editorial content but I would simply say that rights are subject to incremental erosion like anything else not maintained properly and at what point is too much erosion capable of washing away the foundations of the protections such rights preserve.

[webcentric]

As far as I am concerned Google will use structured data as a trojan horse in the future

That is such a better analogy than dogs doing tricks for biscuits. Why have a hoop on my desk when I can have a stone gate in the front yard?

[EditorialGuy]

I agree that using structured data has potential risks in the future. Just like with what Google did with image search, by showing full size images now, it's very clear Google wants to keep people on their properties and not send people to anyone elses website.

That depends on what's being marked up, but in many cases, the correct response to your comment is "So what?" Take a public event as an example:

The organizers of the Widgetville Walnut Festival care less about getting people on their Web site than about getting people to their festival. If a search on "widgetville walnut festival" produces a box on the Google SERP that includes the name, location, dates, opening hours, and ticket prices of the Widgetville Walnut Festival, everybody wins: the festival organizers, people who want to visit the festival, and Google.

Now, if you're the owner of a third-party Web site about the Widgetville Walnut Festival and you want to attract visitors who'll view ads, buy walnut-themed t-shirts, and subscribe to your Walnut Eaters Newsletter, maybe structured markup (or at least Schema.org "event" markup) isn't your thing.

To borrow the old IBM motto, "Think."

[brotherhood of LAN]

blame the browser designers

It is definitely a good point considering a lot of regular users use the address bar as a search bar. I read lately that Mozilla's lion share of revenue derives from Google searches and the search/address bar, so I wouldn't expect them to change the behaviour anytime soon, or put DuckDuckGo as the default.

[webcentric]

To borrow the old IBM motto, "Think."

Think about what? Think about my own personal gain or think about how my actions might impact the world around me. In other words, should I also engage my conscience in the activity or would you advocate thinking more like a bot? Hmmm. Bots with a conscience. How very iRobotic.

[netmeg]

How many of you here are jumping on the structured data bandwagon these days?

No doubt, there are biscuits to be had for jumping through the hoop.

Lol, yep,sell your soul a little bit more.

Yes, there are treats to be had for helping to train the bot

Seriously?

Every time I think this forum might actually be of use, it denigrates to this kind of crap.

[brotherhood of LAN]

I was thinking about locking this thread after some of the Google bashing, please read the focus of the forum [webmasterworld.com] if you haven't already.


Thanks.

[taberstruths]

Every time I think this forum might actually be of use, it denigrates to this kind of crap.

Actually I find this thread interesting. Not because I agree with everything said or the passions expressed but because it shows a greater sense of the disenfranchisement that many webmasters are feeling.

There have been many instances of this happening in history. The end result has been either tyranny or revolution.

Personally I find it encouraging to see that more and more people are waking up and realizing that Google is not webmaster friendly.

Now I totally understand that they are a business and as such they do what is best for them. The problem was that people thought that they were something else. They thought they were their partners or avenue to get visitors to their website. Google itself has tried to give this impression with their PR campaign to webmasters. No webmaster in their right mind gives away traffic for free. Google is no different. So the years of getting free traffic from Google is over.

The question is whether or not webmasters will just acquiesce to the financial tyranny placed upon their businesses or if there will be a revolution that takes place where webmasters tell Google to pay or play with blocking Googlebot from indexing their content.

Revolutions do not start in a day but they are fueled by disenfranchisement.

[EditorialGuy]

Think about what?

Think about what makes sense for you and the user.

It really isn't all that complicated. (See my "Widgetville Walnut Festival" examples.)

[webcentric]

I was thinking about locking this thread after some of the Google bashing, please read the focus of the forum [webmasterworld.com] if you haven't already.

I'm sorry, If you want Google to index your pages, put a meta robots tag in the head of your document with an attribute-value of "index" other wise, use the attribute value "noindex."

Not sure if that's editorial, generally editorial, a rant or instructional. The consequences of your decision apparently aren't up for discussion so you'll just have to figure out for yourself what it all means.

I should also say "thank you for your tolerance thus far" given that it's a season for thanks.

[ohno]

The organizers of the Widgetville Walnut Festival care less about getting people on their Web site than about getting people to their festival. If a search on "widgetville walnut festival" produces a box on the Google SERP that includes the name, location, dates, opening hours, and ticket prices of the Widgetville Walnut Festival, everybody wins: the festival organizers, people who want to visit the festival, and Google.

Google is making nothing from your example other than feeding the info to the user(& perhaps having ads all over the SERP screen). Now, if you seriously think that box on the Google SERP will remain free forever then good luck. Do you remember the Google one box results for products that later became Google Shopping? Remind me what happened there?! Do you think the current Google Shopping results give the best results for the user as Google claim? Personally speaking I liked it more when there was a list or around 20 stores selling the product not how it is now.

[brotherhood of LAN]

I'm sorry, If you want Google to index your pages, put a meta robots tag in the head of your document with an attribute-value of "index" other wise, use the attribute value "noindex."

Not sure if that's editorial, generally editorial, a rant or instructional. The consequences of your decision apparently aren't up for discussion so you'll just have to figure out for yourself what it all means.

That would be instructional and possibly educational to some, and apparently you are correct, as per the terms of service and forum charter. This thread in itself is clearly not about SEO at all and probably should have been posted in Google Finance, Govt, Policy and Business Issues [webmasterworld.com], where the ethical nature of the business can be discussed.

I should also say "thank you for your tolerance thus far" given that it's a season for thanks.

I'm simply trying to abide by the rules set out in the forum charter and terms of service so others can visit this forum and be sure they're going to read about Google SEO, which is what this forum is about. The moderators here are simply volunteers who use the same ToS and charter that you can read, not arbiters of truth.

I'm going to lock the thread now as it seems to have run its course, but you're welcome to start a new thread in the other forum.