Is it time to de-index the Web?

[webcentric]

The Internet is an interconnected network of digital resources. It was conceived as such anyway. Along the way, someone decided it would be a good idea to index all the content on the Web to make it easier to find things and the race was on. The commercial prospects of such an endeavor were just too great to stop people from trying and try they have. Indexing the web is now HUGE business and everyone involved, from single-site webmasters and mom and pop storefront operations to amazon-sized companies and governments on every level all want their piece of the action. Everyone wants to be number one in Google and Bing and Yahoo and, blah, and blah and it doesn't seem to matter that that pursuit is ruining the Internet and shaping the world of information in a very scary way. It used to be that the Internet was a place where moving from place to place was the way to find things. Now, building natural connections between related information is seen as detrimental to the corporations who strive be the only authority on how people should find things on the Web. The result is that these mega-corps are dictating the terms of survival on the Internet when the only end game is truly their own survival. Jump through this hoop today (just to stay occupied) while the magician slices another rabbit's throat and tosses it in the pot.

Webmasters have provided the content that these major corporations are usurping for their own benefit. Without our content, they have nothing but a page of advertisements. Oh, wait a minute, they just built an advertising system to replace the actual Internet and they used our blood, sweat and tears to do it. Good one! Didn't see that one till it slapped me up side the head.

The Internet is on the verge of becoming a portal called Google and soon there will be no chance of promoting yourself for free because all the available promotion space on every page once used for the actual index will be bought and paid for. Indexed content is on the verge of becoming irrelevant and when it does, look out!

So, is it time to go back to networking the Web the way it was originally envisioned so people will actually be able to find things that matter to them in a natural way or are we all ready to concede that advertising is information and it's the only thing the word needs? If so, prepare for a new dark age because soon, there will be no medium left for free expression unless you're ready to chisel it out on a set of stone tablets a couple hundred thousand times and ship it around the world on the wings of a Dodo.

Is it now time to put the noindex value to it's proper purpose. I'm beginning to think noindex is the solution to a very big problem but it's gonna take a hell of a lot of noindexes to get the job done. What do these greedy [fill in the blanks] have without an index? Seems they may already have enough to abandon the pretense of providing search services at any time but I for one would be willing to put that assumption to the test. It's a simple question and a seemingly impossible proposition but I'll ask it anyway. Is it time to noindex the Web?

And I'll ask another question. How much of your bandwidth is Google sucking into it's carburetor to fuel the vehicle which is about to run you down in the street and or back up over you if it's already done so?

[Shepherd]

big players in our field continue to spend with Google

google is a traffic channel, not the traffic channel. A business would be silly not to advertise on google if there is a positive ROI. Just as silly as the businesses that don't pursue other traffic channels like Netmeg suggested.

[netmeg]

Well I guess a fair few others have missed that golden info too as some seriously big players in our field continue to spend with Google. If no one needed them they wouldn't waste money on them.

Um no. That's not how it works. Just because I can bring in traffic via email, direct mail, video, social media, print ads, radio and TV, mobile, other media and press, and various other channels doesn't mean I wouldn't also take search engine traffic as well (paid and organic) They are not mutually exclusive.

[EditorialGuy]

Google can deliver information via whatever format it wants (SERP's, knowledge graph whatever).... nothing changes the fact that they don't own the information. We do

They don't need to own the "information," any more than you do. The U.S. Copyright Office explains:

"Copyright protects the particular way authors have expressed themselves. It does not extend to any ideas, systems, or factual information conveyed in a work."

As for Google's crawling and indexing of the Web, that's called "fair use." And it's a win-win for Google and the site owners whose content is crawled, indexed, and served up in search results.

Now for the important part: Google is not required to send you traffic as a condition of crawling your site. Having your site crawled merely buys you a ticket to the lottery. If you don't want to participate in the lottery, then yes, go ahead and block Googlebot with "noindex" or robots.txt.

While you're at it, you might as well block Bing, Yandex, Baidu, and all of the other search engines, because they aren't required to send you traffic, either.

Also, if you ever write a book, be sure to tell The New York Times book Review, Publishers Weekly, Library Journal, and Books in Print that you don't want them to index or review your book. (Why let them earn money by exploiting your work?) And if a newspaper or news site wants to write a feature about your business, tell them "No," because they might run ads with the article and earn money by piggybacking on your name.

[RedBar]

There have been some interesting posts here, don't let it degenerate into a slanging match. A couple of observations:

1. When was the 1,000lb gorilla downgraded to 500lb? Surely it should be 2,000lb now?

2. These discussions on how to combat Google have been going on for several years now, I'm guessing about 2007/8 was when several of us started to see in which direction Google was heading and sounded the alarms.

3. Only governmental intervention now can stop this behemoth, the US will not, G is far to valuable to them, however will the EU? The EU tends to move very slowly but could spring a surprise at some point. G has to be very wary of the EU but it does have some very strong allies within the institution and whilst G spends money like confetti on new offices it's very doubtful there will be any opposition from the UK!

Meanwhile we have now decided to go onto autopilot, we are closing our web design office all but for two people to maintain our existing sites.

[ohno]

I wonder if you polled 100 people in the UK the following :-
(1) You want to buy product X online tomorrow, where would you go?

How many would answer Google?

If you are ecom most of your sales (traffic) come from the Internet & seeing as Google has over 90& share on that market the two are intrinsically linked IMO.

The fact is certain products rely on search engine traffic & since Google started playing games these sites are finding it impossible to expand. We are surviving on mostly repeat custom, a decade of brand making helps there. But you can't get away from the fact that if Google had not fallen in love with Amazon & ebay over the last two years we would have expanded & employed more people.

[Shepherd]

I wonder if you polled 100 people in the UK the following :-
(1) You want to buy product X online tomorrow, where would you go?

How many would answer Google?

Who's fault is that? If I'm selling "widgets" on widgets.com and people don't say "when I want to buy a widget I go to widgets.com" I failed, period, has nothing to do with google, if people are saying they would go to google instead of widgets.com google did their job, I did not. There's no other way to look at it.

[ohno]

Err, I'm afraid there is. You miss the point that people still stick in "Amazon" & "eBay" for example into the search bar on Google! Have they failed?

[Shepherd]

people still stick in "Amazon" & "eBay" for example into the search bar on Google! Have they failed?

Obviously yes.

[Shepherd]

Why do you think google ranks amazon/ebay so well right now. here's why: google is competing with Amazon/Ebay/everyone for traffic. google wants people looking for things on google, not going to Amazon/Ebay/anywhere else. If people are searching for Amazon on google, google wins, amazon failed.

[matrix_jan]

Imagine if there were 5 major search engines with equal shares of the market. And you had to optimize your websites to satisfy all five. Now that would be a mess...

In today's world, there is simply no way you can get as much traffic from other sources as you get from search engines. Maybe for one or two niche websites, but it won't work for everyone! Search engines get humongous amounts of traffic, and that traffic has to go somewhere, if it's not your website then it will be Joe's.

Whenever I go to the live tab of GA I see few users with keywords of "mydomain" or "mydomain. com" or even "my domain" coming from search engines. They are perfectly integrated with users' online world.

And what makes you believe that if you switch the source (e.g. from google to facebook) there will not be competition and you'll not find yourself on the bottom again?

Search engines are not the issue, competition is. They are just a tool you use to show that you're the best. If you're not the best then tools won't help. Be it google, facebook, bing, whatever... it just won't work.

[webcentric]

They don't need to own the "information," any more than you do.

OK, let's run with that. So theoretically speaking, I could launch a domain. Call it something like ThingsIFoundWhileSearchingTheWebOnSomeFamousSearchWebsite.com

Now all I need to do is copy the results from a search query that I ran in Google, generate a couple hundred thousand pages of links from the results and call it fair use. Is that the idea? If it is, I think it's brilliant. No joke! Hey, theoretically every one of those sites allows indexing and none of it belongs to Google.

Of course, I'd probably have to do this manually because, if you've ever read G's robot.txt file you'll find bots are pretty much verboten. Interesting that the actual robots tag on a SERP is simply

meta content="noodp" name="robots"

[ohno]

Yes, competition is the issue (both in Google's results & search engine share). Listing eBay & Amazon for just about every result on earth is not competition. Ranking them page 1 for everything is a strange way to act if you are in competition with them no? Google doesn't sell anything, Google Shopping is NOT eBay by any stretch of the imagination. Sending YOUR customer to your competitor? How? They aren't in competition any more than we are in competition with Google! Google wants money & the big players will pay that for results, the ads weren't enough so now they have the SERP's.

Anyway, it's been a long day & it's my wedding anniversary so i'm off for some food & drink.

[brotherhood of LAN]

OK, let's run with that. So theoretically speaking, I could launch a domain. Call it something like ThingsIFoundWhileSearchingTheWebOnSomeFamousSearchWebsite.com

Now all I need to do is copy the results from a search query that I ran in Google

I doubt Google would send you a DMCA to be honest. However there is no "value add" in you creating that site, Google obviously has the value of being a search engine.

And fwiw I agree Google can do as it pleases with its own website, like any other business it's required to operate within the law and that's what they do (subject to pending lawsuits of course). I feel a lot of criticism of Google is wasted, use it as a tool for your own disposal.

[netmeg]

Of course, I'd probably have to do this manually because, if you've ever read G's robot.txt file you'll find bots are pretty much verboten. Interesting that the actual robots tag on a SERP is simply

meta content="noodp" name="robots"

Um, all that means is don't list the DMOZ title. Doesn't have anything to do with bots.

[EditorialGuy]

Imagine if there were 5 major search engines with equal shares of the market. And you had to optimize your websites to satisfy all five. Now that would be a mess...

I can remember when the major search engines included InfoSeek, AltaVista, Excite, HotBot, and a few others. And back in the early 2000s, our site got more traffic from Ask Jeeves than it did from Google.

In today's world, there is simply no way you can get as much traffic from other sources as you get from search engines. Maybe for one or two niche websites, but it won't work for everyone!

True. But for some of those "niche websites," the alternatives can be productive. I know a couple who own a tour business. They get most of their business from word of mouth on TripAdvisor.

As for the future of organic search, I think its value for site owners will depend, to a great extent, on what the site owners are offering or selling. I can envision a future where site owners who just serve up quickie facts are killed off by the "Knowledge Graph," but hey, that's evolution. Who gave John Doe or Jane Buck the exclusive right to use National Weather Service forecasts or FAA flight-status feeds? (Those of us who are over the age of 40 can remember when memory managers were expensive add-on DOS utilities and you needed Trumpet Winsock to make Windows work with the Internet. Change happens, and it produces both winners and losers.)

I also wouldn't be surprised to see organic results continue to become less and less visible for e-commerce searches, as Google, Bing, etc. offer more kinds of advertising tools. Web retailers who rely on SEO skills will give way to Web retailers who rely on media-buying skills and customer retention.

Search has been a valuable tool for Internet users since the days of Gopher and Archie. I'm guessing that it will be still around for quite a while--whether not site owners who feel unloved by Google seek revenge by noindexing their pages.

[webcentric]

However there is no "value add" in you creating that site

Well, perhaps I could cherry-pick the results, reorder them, superimpose my own ranking criteria right over the top of G's ranking criteria, explain to folks why these results are better than the ones you get in Google, only show one ad on the page and have a comical video running on the side just to entertain people. Heck I could get a crew of minions to spend all day just cherry-picking through Google results and generating new pages of results on every known topic. Then I could video the minions having fun doing their job and getting a paycheck. Don't need to be found in search because we're gonna link them all together until they're so dense they form a new layer of Internet right on top of the old one and you won't be able to avoid it. Will need to create an index of all these pages of course and a good way for people to hop around. Webrings weren't a bad idea until someone came along and declared the concept detrimental to the future of Google. Maybe I'll use webrings. Google, has killed a great many great ideas in it's monopolistic grab for power. Governmental (or perhaps Divine) intervention may indeed be the only recourse left but I fail to see why sticking one's head in the Kool-Aid is any better than having an original idea.

I feel a lot of criticism of Google is wasted, use it as a tool for your own disposal.

I'd rather look at this thread as an attempt to "think outside the Google box" for a moment. I agree, criticism is pretty much a futile exercise. Weighing one's options is a completely different story though. Envisioning a different path, where's the crime in that? Some want only what they can get today. It's short-sighted but hey, as dangerous as it is, shortsightedness isn't a crime either. There will always be those that argue in favor of maintaining the status quo while others will push at the boundaries looking for something new. It's the way of the world.

[webcentric]

I also wouldn't be surprised to see organic results continue to become less and less visible for e-commerce searches, as Google, Bing, etc. offer more kinds of advertising tools. Web retailers who rely on SEO skills will give way to Web retailers who rely on media-buying skills and customer retention.

You're back to one of my original questions which is essentially, "Is advertising the new information?"

Added: If so, then we have arrived at a point in history where the Internet (that place where so many people in this discussion were able to convert a simple dream and a domain name into a successful business venture) will no longer be the level playing field it once was. This is a story of Corporations against mom and pop. This is conquest pure and simple. Look around your local town and remember all the small, once thriving establishments you're neighbors used to operate and look at the lives that have been ruined by giants stomping around in the neighborhood. Look at the transition of the American workforce for example from a highly-skilled, motivated, dreaming workforce into a service industry. That's where this is headed. Great for everyone who lived in a time in history when you could take essentially nothing and carve your niche on the Web. That time is coming to an end. Say goodbye to your neighborhood grocer. Say good bye to the little guy with the unique idea. The Internet was a new frontier. That frontier is now populated by the high and mighty and there is scant room left on the page for those who would dare to follow their dreams. If advertising is the new information, very few will be able to afford to disseminate it which makes it nothing more than propaganda in the end.

[edited by: webcentric at 5:34 pm (utc) on Nov 26, 2013]

[brotherhood of LAN]

Well, perhaps I could cherry-pick the results, reorder them, superimpose my own ranking criteria right over the top of G's ranking criteria, explain to folks why these results are better than the ones you get in Google, only show one ad on the page and have a comical video running on the side just to entertain people. Heck I could get a crew of minions to spend all day just cherry-picking through Google results and generating new pages of results on every known topic.

Sounds like you're describing a directory.

I think if I can draw any conclusions from this thread, is that search engines are here to stay as a basic human requirement for finding information online... if Google no longer serves that need then someone will replace them.

[JD_Toims]

Sounds like you're describing a directory.

Sounds like that or maybe something like www.dogpile.com to me too.

[webcentric]

Sounds like you're describing a directory.

How true but the point wasn't about form, it was about method.

[EditorialGuy]

You're back to one of my original questions which is essentially, "Is advertising the new information?"

Advertising is "information" for some things, and there's nothing new about that. Long before the Web was invented, people were buying trade and special-interest magazines for the ads as well as for the editorial content. Case in point: Back in the 1980s, PC Magazine averaged 400 pages per issue (the record was 800+ pages, according to Wikipedia), and PC had to go to a biweekly schedule because it had so many ads. And let's not forget the traditional telephone Yellow Pages, where the ads were practically the only information within the directory covers.

For someone who's searching on "model WX-1000 widget price," paid Google Product Research results are as "informational" as the organic listings are. (Maybe more so, since the information is right in front of the user.) Similarly, an AdWord that answers the searcher's question better than a snippet does can be useful information.

Common sense would suggest that, over time, organic results will continue to decline in importance for transactional or commercial searches--not because Google, Bing, etc. are evil, but because well-crafted and well-targeted advertising can be a win-win-win for the advertiser, the advertising medium, and the user. In the history of commerce, advertising has a far longer record of success than SEO-driven promotion does. To quote an old expression, "The more things change, the more they stay the same."

[webcentric]

I hear a theme that says, there can be a balance between corporate dominance of the Internet and freedom of access to real information for everyone else. I guess I'm challenging that notion much the way a Native American would challenge the assessment that pioneers and nomadic tribes can co-exist together on the Great Plains of America and share the buffalo in the process.

[netmeg]

The Internet was a new frontier.

There will be more frontiers. We may or may not live long enough to see them, but there are *always* new frontiers, because people discover or invent them, and other people figure out how to adapt them, and exploit them, and it starts over again.

The internet itself has only been here a blip in time; it was barely even conceivable when I was in school.

[JD_Toims]

Problem: Google's not sending enough free traffic.

Proposed Solution: Noindex the entire web, which will ensure Google and other search engines don't send any traffic to websites and at the same time will destroy the search engines, based on the premise somehow destroying search engines will not only protect site owners, but will also enable people to find the "real information" they're looking for rather than the information presented by sites that can afford to advertise on a search engine's results pages or on one of the rest of the sites in the results.

---

I must be missing something, because if search engines were all "defeated" I can't figure out how people are going to find "the sites with the 'real information' they're actually looking for" in a way that's not already available to them, and why if people are really unhappy with the type of information in the results provided by search engines they keep using one, rather than DMOZ, Yahoo's Directory, calling a friend, looking in the phone book, or some other method they could use right now today if they chose to.

People currently have plenty of choices for finding sites and information other than through a search engine, but the traffic that runs through search engines suggests search engines are the method people use most, so noindexing really isn't going to do much except ensure no one can possibly find the information presented on the noindexed site via the means they use most frequently to find something on the web, which means all someone is really doing by noindexing is limiting their site's ability to be found by people who don't know about it or don't know what information it contains.

[edited by: JD_Toims at 6:37 pm (utc) on Nov 26, 2013]

[mrengine]

I know a couple who own a tour business. They get most of their business from word of mouth on TripAdvisor.

If you are not spending money on Adwords, word of mouth is the only way to survive these days. Google saw to it that now everyone fears links - whether who you link to or who links to you. I would think the anti competitive alarms would be ringing off the hook at the FTC

TripAdvisor is also in that lobbying group Google created last year by the way. You know, the group that says they are the voice of the internet economy. And some foolish people actually believe that too!

[edited by: brotherhood_of_LAN at 6:41 pm (utc) on Nov 26, 2013]
[edit reason] see stickymail [/edit]

[EditorialGuy]

I must be missing something, because if search engines were all "defeated" I can't figure out how people are going to find "the sites with the 'real information' they're actually looking for" in a way that's not already available to them

Here's an idea: A site titled "Spam Me" where the user enters an e-mail address and a topic. Every Web business will have access to the inquiry database, so there won't be any search engines to act like bouncers at the head of a rope line.

[webcentric]

Here's an idea: A site titled "Spam Me" where the user enters an e-mail address and a topic. Every Web business will have access to the inquiry database, so there won't be any search engines to act like bouncers at the head of a rope line.

That's funny. I actually searched for that domain last night while having a similar thought. No joke. It's taken but I found about 10 more related ones that aren't.

[taberstruths]

That would be considered a high value exact match keyword domain. Everybody loves to eat SPAM

[lucy24]

information-only websites

Serious question that absolutely has to be answered if anyone is going to be talking about "knowledge graphs" and similar:

What proportion of the world's information can be expressed in three lines of text?

Are most web searches concerned with learning the distance to Lemuria (two words) ... or with the development of Lemuria's colonial history during the 19th century (many pages)?

Anyone out there have access to a human? Hm, come to think of it, I'm going to bounce the same question off our reference librarian.

If people are searching for Amazon on google, google wins, amazon failed.

Nonsense. Google won, humans failed. People aren't googling for amazon/wikipedia/ebay because they don't know the URL. They're googling because they don't know that there is an URL. There is absolutely nothing amazon/wikipedia/ebay/your-own-site could have done to prevent this behavior. If you want to point fingers, blame the browser designers who have deliberately blurred the distinction between navigation and search.

[Shepherd]

Nonsense. Google won, humans failed

Humans were trained, and google trained them, Amazon should have trained them.

If you want to point fingers, blame the browser designers who have deliberately blurred the distinction between navigation and search.

That is a very valid point.