What comes after Google?

[iamlost]

Among the various development vectors of search cum internet information retrieval there is one that has quite fascinating possibilities as well as quite dire potential consequences for webdevs. The chronic angst as Google adds variations of ‘answers’ is nothing to what is, possibly, just barely, over the horizon.

I’ll use two different category illustrations of what I’m talking about:
1.Current news, i.e. hurricane Florence. Currently, if I want to know what’s happening I may visit one or another news sites, check Google news, etc. However, instead I can send out a number of bots, scrape any number of local news and news agency sites, scrape bounded data, use an algo with or without ML to create a custom news result. From which further specific data can be targeted and specific interests followed.
2.Reviews, i.e. for Widget model FooBar19. Currently, I can do a search and read in serial. However, instead I can also send out a number of bots to scrape reviews of target item and, again using an algo with or without ML, create designed combined results to order.

I’ve actually done the above. And others have done much more. It puts control and context in the hands of the user. And it is all automated. No human involved beyond designating target and parameters and pressing ‘go’ then reading the results and redefining if desired.

As a hobbyist hacker it is heaven. As a webdev looking for revenue from site visitors it is like looking into hell.

As a side issue I’ve been testing just how blatant a user-agent and/or behavior it takes to get blocked. Incredibly many sites, even enterprise, are pretty much wide open. And no sites, even my own with all my years of building defences, catches all the deliberately ‘human’ bots.

Also depending on how it is done it is NOT illegal (in most/all jurisdictions) nor does it necessarily infringe anyone’s copyright. Think of all the chronic angst as Google shifts from search to answer. Now think of Google et al, the middlemen, are rendered moot and visitors act as their own search operators. Instead of Googlebot it’s everyone’s bots.

Where’s the beef?

[Leosghost]

Until such things come pre loaded into smart phones ( and if they do they'll have known characteristics and thus be blockable by webmasters ) this will not be a threat, except to those who do not take the integrity of their sites seriously..

It will also only affect those sites who depend on ad revenue to exist..

Those of us with original unique items in ecom are not going to lose anything to an automated personal aggregator..

You just made the Google hit list..better get back to the yacht and sail for unknown seas.. ;)

[NickMNS]

Discovery?
The bots still need to know where to go. For news it is pretty straight forward as you can feed the bot several news sources/sites but for widgets not yet invented you don't know what you don't know or where to go. Google and others SE's indiscriminately scrape much of the web and then point you the user (or bot) at the sources that are most likely to hold the information/data desired.

This information asymmetry effectively makes Google the gate keeper to the web.

[iamlost]

Whither the gatekeeper when its content is scraped and it's ads ignored...

In a data scrape centric web the major current revenue streams go dry...

Makes for fun prognostication in certain circles...

[brotherhood of LAN]

scrape any number of

You don't need to when the answer to any of the questions is 42.

I'm not quite following your point aside from 'the search engine can eventually provide all the answers and not link out'.

It'd probably help if a search engine admits whether it's 'a tool that searches the web and returns results' and a 'knowledge engine'.

In Google/Bing's case, it seems like there is an ever increasing surface where the search engine can provide the answer to a question via a feed, summary from other's content, or knowledge graph. A lot of time, it is a 'best guess' rather than something definitive IMO, especially for things that have synonyms.

Much of the time to me, it 'makes sense' as a user to see how/why they have some kind of generated summary. Some of it does appear quite sophisticated.

There'll always be a demand for a 'search engine of the web' over a knowledge graph IMO, but maybe the line will stay blurry. If you think it's easy to replicate, 100 grand a month's worth of servers should do it :)

[Mark_A]

iamlost,an interesting theory certainly.

Where in this future would website owners, companies / brands get eyes on their products?

[tangor]

Where in this future would website owners, companies / brands get eyes on their products?

By being unique, authoritative, and better than the rest. For that to happen at least 90% of the copyright copycat sites will need to disappear ... too many possibilities for the "same dang thing".

Given memory (storage) density increasing with costs coming down, it's not that unfeasible to put a search bot on each machine to mine g or b or ddg or yx in the background.

[Mark_A]

I can imagine if the world started to go that way G would be one of the key providers of such bots. And the bots would still have algo updates!