homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque

Webmaster General Forum

This 44 message thread spans 2 pages: 44 ( [1] 2 > >     
Search Engines are Doomed.

 11:24 pm on Jan 28, 2002 (gmt 0)

In one of the better search engine related articles [sfgate.com] I've read in a long time, reporter Hal Plotkin calls out the search engines on paid listings. Hal is famous for an original interview with Jerry Wang [founder Yahoo] in which Yang pledges to keep a separation between listings and advertisements. As readers of this site know too well, that pledge has been shattered.

The problem they [search engines] are now bringing down upon themselves revolves around their misunderstanding of the critical role a very basic journalistic principle plays in running a successful media business.

Put simply, the search and directory firms have put their futures in question by flouting time-tested business practices that require an absolutely clear separation between editorial content and advertising.

That is for the most part getting right to the heart of the matter. It's interesting that most reporters view search engines as "media" companies. That's a tricky distinction for most people to understand. The concept that SERP's (search engine results pages) are editorial content is difficult to comprehend. After all, search engines are robotic mindless thoughtless beasts incapable of "editorializing". It seems to come as a shock to many reporters when they realize that the search engines have transformed from encyclopedic listings of sites (white pages), to paid advertising (yellow pages). The media label dies hard.

There is only one point in the article, I disagree with Plotkin. He points out what is apparently a duality in Googles policy towards urls:

"...we [Google] do not add all submitted URLs to our index.."

Google's advertisers, on the other hand, are invited to buy prominent places on the results pages that appear whenever users search for any particular words or phrases.

I don't believe that characterization is fair. No search engine can add all submitted urls to it's search engine. They can't add them for for several reasons:
a) technical reasons (spider blocks, unindexable content, index size)
b) quality control (spamdexing prevention)
c) copyright, trademark, and legal concerns.

No other major search engine (or site for that matter) on the web today has such clearly delineated advertising as Google does. Sure, I would like to see sponsor replaced with advertiser too, but don't blame search engines for a trend that started with Uncle Milty on TV in the late 1950's. The characterization that Google has a double standard is unfair. They didn't become the most useful search engine on the net by allowing just any old page into the index.

There are a few other technical corrections that need addressing from the article:

>Yahoo! was one of the last big holdouts.

Ah no they weren't, they were one of the first. They were the ones that ushered in the era of paid search engine listings. The order in which search engines "fell":

- Goto.com/Overture has sold results since the first day they went online.
- altavista attempted to sell serp placements in the spring of 1999. There was such a backlash and outcry, that before the summer, they had dropped the program.
- Looksmart began charging $79 in early 2000.
- Yahoo started charging $199 shortly there after.
- Looksmart matched their fee.
- Inktomi began selling spidering services.
- Inktomi later began selling placement via their cloaked feed program.
- Altavista followed suit with their cloaked feed program.
- Somewhere in the above MSN privately began selling placements on their serps.
- Many smaller directories have attempted to follow the Yahoo path. Many have failed.
- In the last two years most major portals including :MSN, Altavista, AOL, Yahoo, and many "meta" search engines have begun using Goto.com results.
- In the last three months, European search engine Fast/AllTheWeb has been beta testing a pay for spidering program.

The search-engine industry is now hanging out in the Internet's red-light district.
It's only a matter of time before most of the public begins to look elsewhere to find what they need.

I don't think it is all that bad. The last year was depressing, and at the same time exciting. We lost some engines, and we had two new engines born in WiseNut and Teoma.

The cost of putting a major search engine online has never been cheaper. Excite was reported to have spent $65 million to develop. That same search engine could be built today for a fraction of that cost and be more powerful at the same time. Those economics are making it very tempting for companies to toy around with search engines.



 11:52 pm on Jan 28, 2002 (gmt 0)

Great post Brett and I agree with all your points.

I think your last point is the most interesting. Will we start to see a plethora of new old school style search engines to replace the erm....old old school?


 2:11 am on Jan 29, 2002 (gmt 0)

Great summary Brett...going along with this:

There's a trend towards monetizing the SERP through:
* Sponsored Listings

Monetizing spidering
* Trusted Feed

Monetizing Directory listings:
* Listing fee
* Yearly fee

As all the major search services decrease in relevancy in order to monetize their business models, surfers will need to begin to look at alternatives. The options they have are:

* Free spidering engines (Fast, Google, Wisenut)
* Niche engines
* Reccommended links from hubs & authorities
* Peer to Peer services (like Napster)
* Lists of bookmarks

Most likely, people will need a starting point..that's always going to be a search engine...then a way to keep going is through links...

Any other ideas as to what people will be using? Thoughts, predictions, other tools you use when surfing?


 2:27 am on Jan 29, 2002 (gmt 0)

I agree 2_much. I was asking my 19 year old son who spends hours on the web how he finds his sites. He originally started with Hotbot, which I thought interesting, and now he just hops from site to site through the site he's at suggested links.

For me it's Google and from there I'm usually looking for themed hubs that are on my topic. From there it's endless.


 3:12 am on Jan 29, 2002 (gmt 0)

paynt has an interesting point there about his son.

We find that you CAN find "what you are looking for" quick using Google, Wisenut, Teoma, and maybe AlltheWeb. But the casual (rather than professsional or regular) surfer which makes up the great majority, don't really know these services with possibly google as an exception. They use MSN and Yahoo, and complain that they cant find anything using search engines.

The concept of trusted sites, who provide external links, is a good one. You find the one or two in your areas of interest and trust those to find what you are looking for throught their own vortal or external links.

I'm also finding some experiments and working examples of listing links by linking popularity as pioneered by the weblogging genre as interesting. Daypop's top links, and Blogdex are two of these. Early times for sure, and limited to that particulalry incestuous and high noise level genre, but there are seeds of good ideas there.


 9:52 am on Jan 29, 2002 (gmt 0)

I don't think all the monetisation that has happened is a bad thing. For instance, I can see where the owners of a very large, rapidly content turnover site would be happy to pay for more frequent spidering. Everyone wins, the site, the SE, and through having better information available, the surfer. I also believe that as the SEs have to make money from somewhere, the advertisers ought to pay. The SEs just havent got the balance right between taking their money, and preserving their core business of searching. Those who do, will win. Those who don't....

Also Goverture has proved that the PPC could work, although there probably isnt a lot of room in that sector at the moment. As the online population grows, along with the level of awareness of the issues, there could be space for others

However, its a new industry, a lot of the people invovled have never been involved with running a big business before, and mistakes WILL be made. Lets just hope that not too many go to the wall


 9:59 am on Jan 29, 2002 (gmt 0)

The author of this article is very harsh towards AskJeeves, is he referring to the Jeeves of old or the new Teoma enhanced version?

Take the financially desperate AskJeeves.com, for example. Most of my recent queries at that site have generated results that appear to be intentionally unrelated to the questions posed. The first thing that shows up when AskJeeves' visitors seek information on the demographics of California's telecommunications industry, for example, is a referral to what looks like a business partner selling consumer-electronics items.

I don't see quite the same results: demographics california telecommunications [ask.com] - though I must say that this is a very tricky query for any SE - may NL's premium search would be a better place to try?


 11:07 am on Jan 29, 2002 (gmt 0)

> c) copyright, trademark, and legal concerns

Hmmm, remember the cache? Or displaying content from pages in the results? It's called infringement...

That's not something they loose sleep over! I'm pretty sure the only way to stop them doing that by default is credible threat of a law-suit ;)


 1:56 pm on Jan 29, 2002 (gmt 0)

It's about time someone began calling attention to the corruption of search results. Responsible journalistic organizations have a clear boundary between editorial content and paid advertising. We've all seen the sleazy advertorial features that blur that line, and it makes us question everything in the magazine; magazines with heavy advertorial content tend to be desperate for any kind of revenue, and rarely become (or remain) major players.

Most search engines haven't figured this out yet. I think Goto was a great innovation, and they were (and are) completely up front about what's paid and what's not - even showing what the advertiser is paying for your click.

Other engines, though, are blurring the distinction between paid an unpaid content, and it will cost them traffic in the long run. People DO tend to gravitate toward better results over time, even the oblivious users we all joke about.


 2:53 pm on Jan 29, 2002 (gmt 0)

I thing Rogerd has called this correctly... although it's frustrating for many of us, waiting for net users to evolve.

I also believe that Goverture KNOW that their lifespan in terms of current revenue is limited. The thread which discussed the Goverture press release regarding relevency checks hinted at this. Essentially, Goverture are attempting to fight the tide by employing stricter intepretations for keyword bidding... PPC though can never really be as relevent as, for example, Google, and they know it.

Yes, Sam Surfer will gradually start to bypass the paid listings, even though some search facilities clearly set out to deceive their customers by presnting them as legit relevency returns.

This latter point is one tha has infuriated me for long time. It is deceipt and it is corrupt... only practised by the dishonest. Time, hopefully, will catch up with these crooks sooner rather than later.


 3:05 pm on Jan 29, 2002 (gmt 0)

> and now he just hops from site to site through the site he's at suggested links.

Mmmm... Seems like that is how I used to do it back in the early days... Better get on with this "themed hub" idea.


 6:32 pm on Jan 29, 2002 (gmt 0)

>>Responsible journalistic organizations have a clear boundary between editorial content and paid advertising. We've all seen the sleazy advertorial features that blur that line, and it makes us question everything in the magazine; magazines with heavy advertorial content tend to be desperate for any kind of revenue, and rarely become (or remain) major players.
Most search engines haven't figured this out yet...<<

Neither have a lot of "old media" magazine publishers. Take a look at Condé Nast's Concierge.com: It's an e-commerce site that uses content from its magazines to hawk travel sales. For example, if you click on "Romantic Getaways," you get a collection of honeymoon packages that Condé Nast is selling under the BRIDE'S Magazine brand. Similarly, modernbride.com markets gift registries, wedding retailers, etc.

For that matter, magazine and newspaper editors have trouble distinguishing between commerce and editorial when they're writing about the Web. In the travel category, for example, they routinely compare sites like Expedia (hardcore e-commerce) with editorial "content" sites. That's like comparing ROAD & TRACK with an automobile dealer.

As far as search goes, I'd much rather have Google run by techies with doctorates in computer science than by a committee of MBAs at AOL Time Warner. :-)


 6:47 pm on Jan 29, 2002 (gmt 0)

As 2_Much says: "There will always be a "Search Engine"!

HOWEVER, I don't believe that SE's will exists as we now know them. It becomes a extremely difficult job to give a user (non-techie) what he/she is looking for when your dB is x-trillion pages big.

If you type "cats +red tail +blue nose" there will be at least, 100 pages returned to you with that search string. Question is, will the user find what they want in the first 10min? If not, frustration sets in and the blame MUST go to "The Web". Or, you type in MP3 and get a bazillion pages back, but what you really want is on page 128,937,291 at the bottom.

That HAS to change and (I'm not a programmer) I don't really see any REAL tools in sight.

Google does it the best right now, but HONESTLY, depends on what you're looking for, it could be a 50/50 shot you'll find it, even on google.

How it will change? Don't really know. For sure Not PPC, people will get tired of Mr. Big always buying his way to the top. Oh and that everyone is listing Overture's top 3, how does that help? Maybe helps Yahoo, but not it's customers..you know the folks we all depend on.

I believe you'll see more Community or topic type sites, and it will support a very specific category and offer real customer related support.

Damn, that was my nickel! lol



 9:44 pm on Jan 29, 2002 (gmt 0)

>>I believe you'll see more Community or topic type sites, and it will support a very specific category and offer real customer related support.<<

Community sites? I don't think so. (Volunteer-based "expert" sites are a hobbyist niche, not a substitute for search.)

Topic sites? Yes, but only if people can find them. (And the quickest way to do that is type a search term into Google.)

Portal-style "networks of sites," a la Suite101.com and About.com? Again, no. That failed model is a leftover from the days of proprietary online services. (It depends on a "network effect" that doesn't exist in the open environment of the Web.)

Fact is, the Web needs honest, open, frequently updated search engines like Google--if only because nothing else has a prayer of keeping pace with with the Web's constant change and growth.


 11:02 pm on Jan 29, 2002 (gmt 0)

<Ducks to avoid incoming projectiles>

What right did anyone have, to expect that the web would be a place devoid of capitalism, competition, and the need for money? We small operators, and those in the SEO business had a field day, using a "free" service for a few years.

Those with a quality site/product, will most likely retain customers. Those who either weren't there to benefit from the "old days", or had less "worthy" sites will suffer. As a businessperson, I/we all need to expect change. These changes are not always welcome, but they come anyway. As surfers start to ignore paid listings <if indeed they do, personally I don't see this happening. People who are acutal prospects to BUY goods or services most likely don't give a rats patootie<--can I say that?> whether an advertiser paid a search engine per click for positioning, or paid an SEO to gain their position, or left it up to the gods of the SE's to place them in high rankings.>

When the SE's get their monetization problems straightened out, there will be more evolution. And anyone who wants to stay in the game, will find a way to "optimize" the money spent. Is that not still optimization? Its not optimization of FREE listings, but it is still optimization.

As one with a journalism background, I really have to question the "journalistic integrity" of a directory, as one would challenge the journalistic integrity of say a phone book. Especially, when we all know fairly well that rankings in SE's AND phone books are not based on literary value.

And the separation of editorial <snicker> and paid advertising? Well, we know that banners whether in the traditional banner position, or skyscrapers, or side buttons are not stellar performers. Any suggestions how to make money operating a search engine without charging the "advertisers?"
I understand the emotions attached to this article/thread, but heck, in the grander scheme of things, how long should a business (SE) stay afloat on its own dime to support all the businesses that benefit from it, and make no effort to sustain itself?

Doing the quick math, there are two sources of income for the SE's. The users, and the advertisers.

Is anyone suggesting that the SE's charge users to use their engines?

Sure, I hate to see my free lunch going away, but there is still plenty to go around, managing the buffet cart for the clients.

At the time Jerry made his pledge, he was also under the mistaken assumption that revenue derived from banner ads would be sufficient. < A LOT of people made that mistake with a brand new medium.> Times change. Economic reality sets in.

Yes, the pledge was broken, yes its all about the money. No surprises - we've got to work with it, or go do something else.


 11:09 pm on Jan 29, 2002 (gmt 0)

I am assuming that Mr. Plotkin is aware that SEO practitioners are paid to "place" their editorial copy with the best positions. In the traditional media he mentions, articles, puff pieces, features shedding light on companies - TRUE "editorial", are not paid in that sense. A Public relations person is not paid for placement of articles, just creation and submission.

SEO is just another advertising service, no more or less "evil" than a search engines struggling to survive by changing the game.


 9:45 am on Jan 30, 2002 (gmt 0)

>charge users

Just like directory assistance!


 10:22 am on Jan 30, 2002 (gmt 0)

Hal's orginal interview with Jerry Wang:


 10:26 am on Jan 30, 2002 (gmt 0)

>> I understand the emotions attached to this article/thread, but heck, in the grander scheme of things, how long should a business (SE) stay afloat on its own dime to support all the businesses that benefit from it, and make no effort to sustain itself? <<

Hmm... the point though is not that anyone grudges SEs making money, it's that some of them deceive to do it.

It's a fine line in some cases I agree, but the nature of a search engine is that it is PERCEIVED to give returns based on relevancy.

This is reality, and as such, to place returns in there that are paid for and then NOT to state that, is deception. Anyone who does that in my book is simply a dishonest crook.


 12:14 pm on Jan 30, 2002 (gmt 0)

Bravo Napoleon,

Paying for services rendered is a reasonable concept. However, it needn't be done at the expense of relevancy or honesty. When LookSmart first began charging for inclusion ... I was one of the first in the queue to throw my money at them. I felt then as I do now that paying for inclusion and even yearly inclusion fees is fair enough ... provided the ROI is sufficient to warrant the expense.

I would gladly pay Google a yearly inclusion fee to remain in their database. I would grudgingly pay Yahoo and MSN and possibly AOL. The dribs and drabs served up by the rest of the wanna be's (sorry Fast) and Overture spawns ... just wouldn't make sense economically for me.

The deception employed by the majority of the remaining SE's and "directories" ... (if they can still call themselves that) is absolutely despicable. As many have pointed out here, the average surfer is unaware that he/she is being hoodwinked with "pay for placement" sites and in some cases, there is little or nothing to distinguish that these sites are paying to be listed at the top. At least AOL have gone about it honestly and identify their paid advertisers. Why haven't the rest? The answer is easy ... they don't want people to know!!! It is intentional deception ... cloaking at its finest!

Proponents of these schemes argue that as long as the surfer finds the product he is looking for ... then who says the site that paid for placement isn't relevant. I acknowledge that while that may be true, I believe that many, much more relevant sites get buried and it is the unsuspecting surfer who is denied the quality and relevancy that they deserve and expect.

There is much money to be made in a "pay for inclusion" business model if executed properly. Why then must SE's sacrifice the integrity of their database with undisclosed, paid ads masquerading as search results?

I have no problems with Overture. They are honest about it and identify their paid listings. Why don't their clones do the same?

Shame on Yahoo, MSN, Excite, Alta Vista, Hot Bot and the rest. They have intentionally set out to deceive their users. It is scandalous at the very least and should be against the law. I am absolutely amazed that the backlash has been minimal to say the least.


 1:20 pm on Jan 30, 2002 (gmt 0)

With all due respect, Napoleon/Liane, I understand your points, quite clearly I think..

>>This is reality, and as such, to place returns in there that are paid for and then NOT to state that, is deception. Anyone who does that in my book is simply a dishonest crook.

Therefore, those who get paid for SEO work are "crooks?" We "fool" the search engines, and the users.

>>there is little or nothing to distinguish that these sites are paying to be listed at the top.

Should the unpaid listings also have a tag that says "Placed by Professional Search Engine Optimizers" when we are indeed paid to provide this service? Or do we maintain the perception (illusion/deception) that rankings are granted to the deserving by some beneficent being at the search engine?

The question therefore becomes merely who is paid, and does the end user truly suffer harm from either. Of course, the hard working webmaster or business person pays for a service, but do they really suffer?

If the user suffers, whether the paid listings are indicated or not, the effect ripples down to how many users use that search engine. Regardless of whether they know if a listing is paid or not, if the listing results do not satisfy their needs, they will move on.

It all sounds to me like we are basically bitter that we are moving toward paying for what we used to get by being clever.

...This I don't quite understand:
If the SE's are paid, they are sneaky deceptive animals if they don't declare that the listings are paid. SEO's who put listings in their index, and are paid for the service of gaining high rankings, are immune from declaring their commercial gain, and their attempt at "hoodwinking" the user, and the SE, and are above reproach.

Having the user pay for search is an option, however, I think most of us would agree that if a user was charged for using a search engine, they'd just move on to the next free one.
(Like directory assistance... I refuse to pay for that.. I use the (so far) free resources of the web. Good analogy though, Mike.

PS. Personally, I will remain as "evil" and "deceptive" yet ruthlessly relevant as necessary, until the ROI on my SEO efforts fall below that required to maintain my income.


 1:36 pm on Jan 30, 2002 (gmt 0)

Personally I do not care for the Overture affiliates (Yahoo, MSN, ALtaV, etc). I advertise for getting the odd refferal from the uneducated searchers.

I only care for Google and it's possible look-alikes (Teoma, Wisenut)and Allthweb. I am sure Yahoo and others will go down the same spiral as AltaVista did last year. Google's marktet share is rising steadily everywhere.

The main issue is how profitable does Google want to get? They are already profitable (so they claim) with Adwords and selling their engine for intranets and I am sure Adwords is going to get them much more business in the near future. But can they resist the other temptations? (paid spidering, become a search member etc.) Yielding to any too commercial temptation which will give an unfair competitive advantage to cash-richer companies, will help Teoma or Wisenut in the saddle and everyone will be happy with some healthy competition.

The best thing that can happen to Google would be Microsoft coming with something good and big, before people look to Google as the next Microsoft.


 1:41 pm on Jan 30, 2002 (gmt 0)

>> ...This I don't quite understand: <<

Maybe you don't want to?

We create quality sites full of good content... the SEs rank the sites based upon the relevancy of this. If SEs focused upon algorithmic improvement, they would force SEOs to focus upon quality. Result... pretty much everyone wins.

This is the route Google has taken. Spamming and/or SEO tricks are becoming less effective. The best way to get high ranking is to focus on content and quality. It is a natural progression. It is evolution and not deceptive at all.

PPC/etc is counter to this. It doesn't force sites to evolve positively for ranking at all. Quite the opposite. It is the quick fix for the rich.

Worse still, in most cases, it is the deceptive, dishonest and crooked quick fix, as explained at length above.

Whatever gloss you choose to paint, taking adverts and not displaying them as such is crooked.


 2:43 pm on Jan 30, 2002 (gmt 0)

> Having the user pay for search is an option, however, I think most of us would agree that if a user was charged for using a search engine, they'd just move on to the next free one.

Timing is everything. If the Net really goes down the tubes per some of the scenarios mentioned here, I suspect there would be a few million people who would pay a small annual subscription fee to Google to get quality results.

On the other hand, if they are getting their needs filled elsewhere, they wont. Goverture will continue to satisfy their needs for a Yellow Pages. If they need something else... e.g information... and cant get it, history shows they will pay for it (e.g. newspapers, magazines)


 3:23 pm on Jan 30, 2002 (gmt 0)

I think the print media learned its lesson long ago. Legitimate media organs make a real discernable distinction between editorial content and advertising.

The time will come when Joe Surfer catches on that the search engines are deceptive in their failure to disclose the distinction. Building trust is essential to sustained growth, and those SEs that don't heed that principle are bound to fail.


 3:33 pm on Jan 30, 2002 (gmt 0)

My point exactly, Go60... it's just tedious waiting for the darwinian effects to kill off the sellouts.

There's nothing wrong with ads - we all know of magazines that people buy PRIMARILY for the ads, not the editorial content, and many others where the ad content is of equal interest to the editorial content. What causes problems is the lack of distinction between the two. You wouldn't want to read a glowing review of a home theater system, only to find out (or NOT find out) that it was written by the manufacturer's PR department.


 3:44 pm on Jan 30, 2002 (gmt 0)

As long as the search results are accurate people do not really care. It seems that the paid listings find there way inappropiatly into the results for the engines that use them. That is why most of the people I know use Google exclusively. Any ads the appear with the search results are not generally deemed to be offensive but rather expected. Look at the ads in your newspaper. People look at it them much the same way.


 3:57 pm on Jan 30, 2002 (gmt 0)

Wouldn't it be great if Overture went down and the search engines were back to where they began? Back to the way a true search engine should work? Rankings based on quality content, not how much you pay? (Speaking hypothetically of course! But we all know that money pays the bills!)

Is there any way to find out actual statistics of who clicks on these paid listings? What percentage of people just scan below these listings to the web search results?

Robert Charlton

 6:12 pm on Jan 30, 2002 (gmt 0)

>>Whatever gloss you choose to paint, taking adverts and not displaying them as such is crooked.<<

I couldn't agree more... and this blurring is for me the crux of the problem. There are many shades of grey in this, though, what with Inktomi's excessively slow inclusion of free sites, or it's in-house cloaking program for large clients, for example.

What has happened with the Nader group's petition to the FTC?


 7:02 pm on Jan 30, 2002 (gmt 0)

> discernable distinction between editorial
> content and advertising.

I can agree with that, but there are fuzzy lines in some shows:

- Paid product placements in tv shows and movies (Such as John Travolta reportidly taking hundreds of thousands to smoke in his movies, or the blatant promotion in Waynes World).
- Movie stars getting paid big money to do the "talk shows" like Leno and Letterman to promote movies (infomercials).
- Product announcements on the morning "news shows". Kattie Curic has been call the best pitch woman in the business for all the products they parade across the screen.
- Backdoor tie-ins. Take NBC again who is owned by GE, who is owned by etc, etc. You look at some of the products that show up in shows and it is pretty clear to me there are tie ins to product "families".
- The news itself is highly tainted with products. Watch some of the health reports, and you'll get a creepy feeling that the story was for a product and not about the disease.

The line between editorial content and advertisments has never been more gray. I can understand how the search engines could become confused over it.

This 44 message thread spans 2 pages: 44 ( [1] 2 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved