Creating alternative to Google serps

[EditorialGuy]

---

System: The following 6 messages were cut out of thread at: http://www.webmasterworld.com/google/4690067.htm [webmasterworld.com] by goodroi - 9:17 pm on Jul 23, 2014 (utc -5)

---

Need in-depth research information about a product? Google won't provide it so searchers have to go elsewhere.

Yes, and in most cases, they go elsewhere by clicking on Google search results.

[rish3]

Let's create a new engine that doesn't have it's own tech, but rather, scrapes Google, moves the ads back to the right sidebar, demotes amazon, demand media, etc, and sends the results back to the user.

A database of just SERP diffs is easier for a startup to fund, and the delicious irony when Google sues you for co-opting "their" content and presenting it as your own (hello KG!) is totally worth the consequences :)

[superclown2]

Let's create a new engine that doesn't have it's own tech, but rather, scrapes Google, moves the ads back to the right sidebar, demotes amazon, demand media, etc, and sends the results back to the user.

You wouldn't even have to do that. Just produce a free app which analyses the SERPs, reduces the ads and other debris and demotes the brands. The SERPs would be so good that 'New Google' would soon have 100% of the traffic.

[EditorialGuy]

Let's create a new engine that doesn't have it's own tech, but rather, scrapes Google, moves the ads back to the right sidebar, demotes amazon, demand media, etc, and sends the results back to the user.

You may be swimming against the tide:

[searchengineland.com...]

[rish3]

You may be swimming against the tide:

Ah, well, it was tongue in cheek.

But, since you mention it, they scored an 83.

Good vs Bing and Yahoo, but is that really the comparison for excellence in online experience?

Compare that with some other companies for the real context. Amazon.com got an 88 last year.

[superclown2]

Seriously though; I do wonder what the legal situation would be if someone altered Google's SERPs without their permission. All search engines, not just Google, drove a cart and horses through copyright laws and Big G is at the forefront of continuing to push the accepted boundaries further and further out. So what would happen if someone breached their (I assume) copyright on the search positions of sites whose copyright they themselves are breaching? Sounds like a real money spinner for the lawyers.

[philgames]

I would use that app :-)

What if theoretically if a search result only showed 1 site on google
and this 1 site then decided to display on their own site the same search results that was on google would that mean google could sue the site? (I wouldn't put it past google to try)

Also what about an open source search engine that pays people for sharing their data and crawling the internet for the search engine?

[jmccormac]

Google throttles bot queries from what I remember.

Google can be defeated at a country market level, especially where there is a ccTLD that does not publish new domain registrations.

Regards...jmcc

[not2easy]

Let's create a new engine that doesn't have it's own tech, but rather, scrapes Google...

There are already a few doing this, ixquick a duckduckgo come to mind. They don't run their own bots, but reorganize the other engines' serps into what many see as old style G results. Yes, there are ads, you barely notice them.

(Almost forgot - at least one of these - ixquick - does not track your search activity)

[edited by: not2easy at 2:05 pm (utc) on Jul 24, 2014]

[jmccormac]

I think DDG runs spiders.

Regards...jmcc

[CaptainSalad2]

I really love the idea of an open source not for profit search engine! Open source programmers are just as if not smarter than the highly paid staff at google! Best of all the to it for job satisfaction rather than profit (a foreign concept to many)!

That is something if get behind, the whole online community (excluding the stockholders) could get behind, a real grass roots movement! I'd be happy to slap fee advertising for it on all my sites to get the word out as I'm sure many others would!

Wow imagine that worked and all ad money was used to find charities and worthy human research?

I guess if someone was bold enough to start such a movement they will need a company slogan, how about 'back to the internet' or 'don't be google'? ;)

[jmccormac]

Sounds great. Who pays? :)

Regards...jmcc

[rish3]

Found a little more context on that ACSI customer satisfaction score that EditorialGuy raised...

Here's Google's Historical Scores:

 Year 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 
-------------------------------------------------------------------- 
Score 80 | 82 | 82 | 82 | 81 | 78 | 86 | 86 | 80 | 83 | 82 | 77 | 83 

(See: [theacsi.org...] for more detail.)

The peak scores were in '08, and '09, followed by a dip, a partial recovery, a dip, and another partial recovery. If you gather dates for significant events, you can make some guesses as to causation. Certainly on the list would be:

The point at which SERP manipulation peaked, prior to Panda, Penguin, etc. (personally, I think that's 2010)

Some success in clearing out spam with various Panda/Penguin rollouts.

Some perhaps overzealous Panda/Penguin rollouts that created collatoral damage.

The increasing aggressiveness of ads on the SERP pages. Several events there.

On that last point, there are some historical screenshots and analysis of Google's "land grab" in a series of blog posts on rimmkaufman.com. I know posting links here is a subject of some controversy, but this particular series is really good:

[rimmkaufman.com...]
[rimmkaufman.com...]
[rimmkaufman.com...]

[jmccormac]

Jimbo Wales tried to launch a search engine that would have people contribute time and expertise in the same manner as Wikipedia. The catch was that Jimbo and his business partners would get rich from this time and expertise.

The hardest part about building a search engine is not building the search engine. It is building an index and keeping it clean. I don't consider Google's anti-spam efforts to be effective. The problem with Google is that 95% of its spam problem is self-inflicted. Those who built the initial wikiasearch attempt at a search engine were quite good at the search engine side of things. (It was Nutch based.) However, like most people without real world search engine development experience, they didn't understand the problems of the blind crawling approach until it was too late.

Regards...jmcc

[philgames]

Could you do some sort of bitcoin approach? Ie browser app that earns you money for surfing the web.

Bitcoin goes to show how things are possible its just finding a system that works.

I wouldnt be such an angry person and would be more accepting of decreases in my rankings if I knew it was part of a system that wasnt googlistic.

[jmccormac]

This is part of one post on the old Wikia Search mailing list that covers a lot of the topics connected to starting a search engine:

"Ideally, the best resource would be more time. By comparison everything
else pales. The holy trinity of search is bandwidth, hardware and
software.

The bandwidth required to spider tens of millions of websites on an
ongoing basis is considerable. Therefore such a venture would need a lot
of available bandwidth.

The hardware is also a very significant requirement. It would need a lot
of servers to do a proper crawl of the web. It would also require a
backend to process the resulting data into something usable. And a
search interface would be required.

The software aspect is perhaps somewhat easier as the task can be
clearly defined. It has to be scalable, fast and provide good results.
However that is a massive simplification. There are some good Open
Source products out there that do the job well. Nutch is one of the most
popular products in this respect. It also has the elements of
scalability required for large indices. And the tools to work on the
resultant data are well developed and supported. Most of the work will
be on the resulting data.

The search index is the hard part. It takes a long time to develop a
good, clean index. The Infinite Monkeys approach to building an index
(following links and hoping that they will lead to new pages) is not the
most efficient method of building an index quickly when any of the prior
requirements are absent or deficient.

A good index makes the difference between a great search engine and a
spam infested pile of junk. I'm not convinced that the Wikia people
quite appreciate the level of work that goes into that aspect of
developing a search engine. Crawling a clearly defined index such as
that of Wikipedia or some other silo site is easy. However crawling the
web is like trying to take a slice of a swirling nebula.

It isn't really a question of what we want. It is more a question of
what the Wikia project can provide to make the task of developing a
search engine easier. Developing a viable search index is the hardest
task of all - the other elements (the hardware, the bandwidth and the
software) can be acquired to some extent.

So what exactly can Wikia offer? Bandwidth? Hardware? Expertise? Can you
give us some descriptions and specifications of the resources and
expertise that is available to search engine developers? For most of us,
we have to deal with the realities imposed by hardware and bandwidth
limitations. We don't have the luxury of just theorising - everything we
do is geared towards survival in a highly competitive market. Perhaps we
SE people really are on a different wavelength to the Wikia people.

Perhaps the question foremost in the minds of many of the SE people on
this list is this: why should we provide the search expertise? Or, to
put it less diplomatically, why should we make you rich?"

Regards...jmcc

[EditorialGuy]

I really love the idea of an open source not for profit search engine!

Right. And who'd dominate the open-source contributions? SEOs and marketers. The inmates would be running the asylum.

The hardest part about building a search engine is not building the search engine. It is building an index and keeping it clean.

There's also the question of how such a search engine would be financed. Infrastructure isn't free.

[jmccormac]

Do you bother to read the contributions from others? The "who pays?" question is clearly visible above.

Regards...jmcc

[rish3]

who'd dominate the open-source contributions? SEOs and marketers. The inmates would be running the asylum.

Heh. Which current search engine isn't being run by marketers?

[CaptainSalad2]

>>Right. And who'd dominate the open-source contributions? SEOs and marketers. The inmates would be running the asylum.<<

Worried?

[EditorialGuy]

Which current search engine isn't being run by marketers?

Are you suggesting that the search engineers at Google, Bing, Yandex, etc. are marketers?

The "who pays?" question is clearly visible above.

The question is, but I don't see any solid or convincing answers.

[CaptainSalad2]

Who pays? Obviously initial funding is a problem but once off the ground it pays for itself and is self sustaining! Initial sponsors wouldn't be that hard to find and I'd happily contribute to something with substance, 10000 web masters contributing to save their own future?

As for spam if you were not for profit just imagine how many human editors you could employ for 400 billion per year ;)

[jmccormac]

Are you suggesting that the search engineers at Google,

Well they do just shove adverts on high value keywords while crowding out organic SERPs these days.

Regards...jmcc

[jmccormac]

As for spam if you were not for profit just imagine how many human editors you could employ for 400 billion per year

Employing human "editors" isn't a real search engine developer strategy. That's a content farm strategy.

Regards...jmcc

[rish3]

Are you suggesting that the search engineers at Google, Bing, Yandex, etc. are marketers?

I'm suggesting that they are all marketing-led endeavors.

Or, to quote someone more knowledgeable on the topic:

we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.

[CaptainSalad2]

Yea I was kinda joking! If you had 400 billion per year to spend all on improving a search engine rather than shareholders we all know where search would be by now, perfection! ;)

[webcentric]

Why do you folks start the interesting conversations when I'm about to head out on the road for a long weekend? Oh well. I've had some ideas on this subject for quite awhile now and even have some application prototypes floating around somewhere on my hard-drive.

My primary thought is that a search engine doesn't have to cover every topic on the web. It could, for example be restrict to a topic or locality. Constraints like those present some interesting possibilities.

[jmccormac]

Vertical search engines can be very effective. Google's GIGO approach doesn't work well in comparison to vertical search.

Regards...jmcc

[iammeiamfree]

I think it would be good if a webmaster could build a mini index for their site and be able to serve results for all the likely queries and it might even be possible to pull results from other engines when the query is unusual or redirect to a results page on another engine. The mini index would be something the webmaster could manage and incorporate some of the best links directly in their webpages. If you think about something like arelis which is a link directory software it works good for making connections but imagine how powerful it would be if you could make relationships with the sites in your index like email and say I have chosen to add you to this webpage and you are number 1 for this query in my engine. Can we work to exchange traffic for mutual benefit etc. Then once this sort of thing is set up and takes off you could just pool the data from all of the participants. Say there are 10 people from some of the main sites in a niche using that data you could serve results for those queries using a mix of those peoples indexes. With loads of webmasters involved in making quality mini indexes and promoting their engines everyone would have an interest in marketing the overall engine to the public and with the support of webmasters it would be likely to become very popular. Obviously needs a lot more thought but I think the idea of starting with something small and powerful that can be easily achieved is a good one rather than trying to start out trying to index the whole web which is obviously a huge cost and difficulty. What would probably happen is the spammers would also set up their mini index and put their site in top spots but because the better sites didn't have the spammers site in their index the ranking for the spam site would be lower. You would allow your competititors a highish ranking in your index because their sites are ok and you want to help your visitors and not look like you are being unfair so it could actually work to make quite a good and fair serps on the overall engine.

[edited by: iammeiamfree at 7:11 pm (utc) on Jul 24, 2014]

[EditorialGuy]

OK, let's look at some practical issues.

1) Where does the underlying data come from? If you're licensing a feed as a Google "search partner," that's one thing, but if you're talking about scraping Google's search results, which set of Google SERPs do you scrape?

2) How do you correct for "garbage in, garbage out"? Let's say that Google is ranking scrapers higher than the original sources, or that John Doe's site has disappeared from the SERPs because of a wrongly-applied manual penalty. Are you going to write a complicated algorithm to run on top of Google's already complicated algorithm? Are you going to hack into Google's data centers to extract pages that aren't being shown? And who decides whether the penalty against John Doe was legitimate or unfair from a searcher's point of view?

3) You talk about "demoting Amazon" and, presumably, other brands that compete with your search engine's sponsors. Is this idea driven by what searchers want or what you want? And who decides which sites should be "demoted"? (Not the sponsors, presumably, or Amazon could just kick in a few million bucks and demand a spot at the top of the heap.)

4) Where is the demand for this "alternative Google" coming from? Duckduckgo certainly hasn't taken the world by storm, and Bing hasn't been able to take market share from Google despite huge expenditures on search technology, promotion, and advertising. Is there an audience for an "alternative Google" beyond disgruntled SEOs and site owners?