Half of the human population is of below-average intelligence. They will be that dumb. Guaranteed. And safeguards generally only get added until after someone notices that a wrong answer is, in fact, wrong, and complains.

In part, I believe someone’s going to die because large corporations will only get serious about controlling what their LLMs spew when faced with criminal charges or a lawsuit that might make a significant gouge in their gross income. Untill then, they’re going to at best try to patch around the exact prompts that come up in each subsequent media scandal. Which is so easy to get around that some people are likely to do so by accident.

(As for humans making up answers, yes, some of them will, but in my experience it’s not all that common—some form of “how would I know?” is a more likely response. Maybe the sample of people I have contact with on a regular basis is statistically skewed. Or maybe it’s a Canadian thing.)

if you even ask a person and trust your life to them like that, unless they give you good reason they are reliable, you are a moron. Why would someone expect a machine to be intelligent and experienced like a doctor? That is 100% on them.

Insurance companies are already using AI to make medical decisions. We don’t have to speculate about people getting hurt because of AI giving out bad medical advice, it’s already happening and multiple companies are being sued over it.