Forum Moderators: open

Message Too Old, No Replies

Accuracy of AI Summaries: What have you Discovered?

         

engine

2:42 pm on Jan 6, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



As part of my research, over the past three weeks i've been assessing the accuracy of the AI feed in Google, and i've not been impressed.

Around 30% of the output was highly questionable, with around 10% of that being wrong, inaccurate, or incomplete. The rest was ok, with the exceptions of around 10% correct.

A young user, searching for a topic could easily get the inaccurate which sways their thinking.

It's just not good enough for prime-time, imo.

I've just started a similar assessment of Bing but have not yet got far enough with the research.

phranque

3:56 am on Jan 7, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



since i'm going to have to fact check it anyway i've always skipped past the AI overview.

[edited by: phranque at 7:59 pm (utc) on Jan 7, 2025]

Brett_Tabke

4:45 am on Jan 7, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I skip Googles, but have been totally impressed with SearchGPT/ChatGPT search (whatever we are calling it). It has been my default engine now for a few months and I rarely head back to Google (only for maps and youtube), or Bing for images.

engine

4:58 pm on Jan 7, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I wish I had access to SearchGPT, Brett. I was on the wait-list, but didn't get access.

ronin

5:43 pm on Jan 7, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



@engine

[forbes.com...]

Featured image: webmasterworld
www.forbes.com
ChatGPT Search Now Free For All. Here's Why You Should Try It
OpenAI announced Monday that ChatGPT Search is now available free for all users. It was previously only available for paying users behind a subscription paywall.

Kendo

3:02 am on Jan 8, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I have used AI to write articles on my own specialty and found that it used my other articles as notable references.

Does that make it good?

explorador

9:20 pm on Jan 8, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




Accuracy of AI Summaries: What have you Discovered?

Wrong and false data/info, even non existent stuff.

I've been doing editorial work for clients regarding their projects (diff nature). Some stuff I already know, but sometimes I need to search for information in order to provide a comprehensive background of information for them to understand, and this also requires sources (with reputation), because some pieces of data may or will create debate with other authorities. Some stuff... I don't know, it's new to me, but I know the process of finding sources with good reputation (at least 3 of them), read, check, compare, and even check those sources to find out who they are or what they do, etc.

During this process, I've found on AI summaries (on Google) stuff totally made up.

Due to the fact that most of my work happens in my native language (spanish), it's funny to see made up words, or translations from english concepts/words, that don't exist in spanish. I don't remember right now, but I came across a word that doesn't exist, being said as valid, and it's meaning being also described, source? RAE (Real Academia Espaņola), that was fun, and when you check the information (regarding the sources) there was absolutely no link to the RAE.

I happen to have knowledge about my own languate (spanish) way beyond the average user/writer/reader, so I find this both funny, and shocking. It's a problem, because many MFA websites have been created using AI stuff, and then, students use those "word meaning" websites as sources to back up their work (even university teachers have been doing this), and this propagates FALSE information.

The problem is having people telling you it's true because it appeared on Google.

Go figure.

explorador

3:47 pm on Mar 14, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



By the way:

AI search engines cite incorrect sources at an alarming 60% rate, study says

[arstechnica.com...]

Featured image: webmasterworld
arstechnica.com
AI search engines cite incorrect sources at an alarming 60% rate, study says
CJR study shows AI search services misinform users and ignore publisher exclusion requests.

Whitey

7:40 pm on Mar 15, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Since this rate of inaccuracy is unacceptable, I'm wondering how or if it will be addressed.

Traditional SERP's don't get it right either, but the user mindset has adapted to this, which served Google well in driving Ad sales while users poke around extensively trying to refine their search. I wonder if users will slip into a similar malaise with AI and if AI search engines will find that "sweet spot" of acceptability that suits their commercial purpose tbd.