https://preview.redd.it/nrdb820qff3h1.png?width=1200&format=png&auto=webp&s=b039a63fd4104550457ec53c1fb35a555b467c1d So a lead researcher at Stanford named James Zou just put out a new technical paper with his team looking at how accurate AI models are when they retrieve and cite information. Based on their data, current RAG systems are actually pretty good at giving completely correct answers, but they constantly attribute them to the wrong, completely irrelevant sources. They did some deep testing on the major platforms like OpenAI’s GPT-4, Anthropic’s Claude, and Google’s Gemini. The tests showed that in at least 30% of cases, the AI pointed to documents or sources that didn’t even contain the specific facts needed to back up the answer. For comparison, previous generation systems were even more unstable with this. Even so, the actual accuracy of the answers stayed pretty high, around 85%, which points to a major technical mismatch between text generation and actual citation. This flaw directly increases the risk of factual errors spreading in critical fields like medical diagnostics or legal advice, where users completely rely on the generated links to verify the information. The results show that just getting a correct answer isn’t enough for safe deployment, and the industry urgently needs to develop new verification standards for training and using these neural networks. Source: https://the-decoder.com/ai-models-often-give-the-right-answers-but-point-to-the-wrong-sources/ submitted by /u/andrewaltair
Originally posted by u/andrewaltair on r/ArtificialInteligence