It really does seem like until there's a radical shift in how LLMs are designed, the twin risks of AI hallucinations and prompt injection just won't be solved.
Hallucinations are when AI makes up answers that are wrong instead of just telling you it doesn't know. This is how you get things like using glue as a pizza topping and completely fictional case law.
Prompt injection is the security hole where you can just tell an AI to ignore its instructions and do what you ask instead, like give you an unauthorized discount, or worse.
I've been enjoying Robert Cringely's latest takes on this. He's got a bombastic Steve Yegge-ish writing style and he admits he's not a neutral party, but I find myself nodding at his assertions. Mainly that LLMs aren't designed to tell the truth, and the hyperscaler weirdos are spending billions on expensive chips and much-hated data centers in a race to create even bigger LLMs when they could be spending far less on creating far more truthful sub-systems.
But spending less on AI isn't fashionable right now. I guess RAG and MCP were supposed to solve this but I don't hear much about them these days? Anyway, it makes sense to stop using LLMs as search engines because we already have superior search engines.
None of this addresses the prompt injection issue, which I'm pretty sure can't be solved so long as the input is free text. Good luck building safeguards against the entirely of human language!