For OpenAI, o1 represents a step toward its broader goal of human-like artificial intelligence. More practically, it does a better job at writing code and solving multistep problems than previous models. But it’s also more expensive and slower to use than GPT-4o. OpenAI is calling this release of o1 a “preview” to emphasize how nascent it is.
The training behind o1 is fundamentally different from its predecessors, OpenAI’s research lead, Jerry Tworek, tells me, though the company is being vague about the exact details. He says o1 “has been trained using a completely new optimization algorithm and a new training dataset specifically tailored for it.”
OpenAI taught previous GPT models to mimic patterns from its training data. With o1, it trained the model to solve problems on its own using a technique known as reinforcement learning, which teaches the system through rewards and penalties. It then uses a “chain of thought” to process queries, similarly to how humans process problems by going through them step-by-step.
At the same time, o1 is not as capable as GPT-4o in a lot of areas. It doesn’t do as well on factual knowledge about the world. It also doesn’t have the ability to browse the web or process files and images. Still, the company believes it represents a brand-new class of capabilities. It was named o1 to indicate “resetting the counter back to 1.”
I think this is the most important part (emphasis mine):
As a result of this new training methodology, OpenAI says the model should be more accurate. “We have noticed that this model hallucinates less,” Tworek says. But the problem still persists. “We can’t say we solved hallucinations.”
Why does ai that has a “reasoning” step become dangerous?
assuming that “AI” has “reasoning” and using it in applications that require that is dangerous
Sorry but thats not an explanation of your position, thats restating what you just said.
It will be used to take control over peoples lives.
In any simple way it may be - denying job/insurance/care/etc, it will be hailed as using ‘reason’, while it just repeats patterns from the training sets.
It does not ‘reason’, because it can’t. Trying to sell it as such is very dangerous as it will be used against people, and it’s dishonest for the investors as well, as they will jump on it even though it’s not ‘true’ and it never will be for this model.
https://softwarecrisis.dev/letters/llmentalist/
I agree that is a bit of an ethical minefield to employ it to make decisions that affect peoples livelihood. But my point is if a company uses it to decide if an insurance claim should be paid out, the models ability to make those decisions isnt changed by what we call the steps it takes to come to a decision.
If an insurance company can dissect any particular claim decision and agree with each step the model took, then is it really different than having someone do it? Might it be better in some ways? A real concern is the fact that ai isnt perfect and mistakes made are pretty hard to accept… seems pretty dystopian i get that. But if less mistakes are made and you can still appeal decisions then maybe its overblown?
LLMs just repeat training sets - so every mistake is repeated forever.
Every bias is locked in and can’t be fixed.
So you just deny people and expect them to appeal everything… sounds like you are offloading costs on the victims.
Shit like that is what makes it demonic.
I think its the company’s responsibility to incorporate a technology to carry out their policy accurately. They cant just use an LLM stock from a vendor. They work to adapt it for their needs and get acceptable results. I think if an llm isnt considerably more accurate than humans then its a disservice to their customers and they should be responsible for that. There should be regulations to keep companies from using models if they dont work