A new search engine needs an index of the web. And many sites don’t welcome any web crawler that isn’t Google or Bing.
engine
11:48 am on May 26, 2022 (gmt 0)
That's a minor hurdle, imho. If it was a trustworthy search I think it would just need time to build a database. It took google years to get as far as it did.
Brett_Tabke
12:07 pm on May 26, 2022 (gmt 0)
So Neeva, wants to build and index, then put that index behind a paywall and charge for it? What do webmasters get out of it?
lucy24
2:38 pm on May 26, 2022 (gmt 0)
You get visitors who are willing to pay for something they can easily get for free. In certain types of business, that is surely the clientele of your dreams.
Brett_Tabke
2:59 pm on May 26, 2022 (gmt 0)
But to start, there is zero traffic and only the vague promise that some day there 'might be' traffic. And if they are backfilling with Bing anyway, why would I let their crawler abuse my site?
Subs just aren't going to work in the vast majority of instances, primarily because most people just search but don't even think about the consequences.
brotherhood of LAN
9:27 am on May 27, 2022 (gmt 0)
I thought Brave's statement of abiding by Googlebot's robots.txt rules and using data centers with an unidentified UA (anecdotally, a Chrome UA) would be the most contentious for webmasters. Understandable why they would of course, given the article.
Facebook is probably one of the best examples, they do allow a good number of crawlers in but all other unnamed UAs are locked out.
tangor
10:00 am on May 27, 2022 (gmt 0)
The day search engine wannabees pay to index a site things will change. Until then we're stuck with what we've got. Sigh.
Brett_Tabke
12:05 am on May 28, 2022 (gmt 0)
> search engine wannabees pay to index
Hey, Google is paying news sites for content - anything is possible.
