- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.
Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.
The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.
Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.
These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.
In every way? How about speed? The goal is to save human time so if AI is faster and the summary is good enough, then it is a success. I guarantee it is faster. Much faster.
A summary is useless if it’s not accurate.
If you make enough mistakes, speed is a detriment not a benefit. Increasing speed allows you to produce more summaries but if you still need to correct and edit them all you’ve done is add a step where a human has to still read the document to the level where they could summarize it and edit the AI summary. Therefore the bottleneck of a human reading the document and working on a summary is still there. It would only potentially make it slightly easier if the corrections needed are small and obvious.
47% is a fail. 81% is an A-… Sure the AI can fail faster than a human can succeed, but I can fail to run a marathon faster than an athlete can succeed.
I guess by the standards we use to judge AI I’m a marathon runner!
I’d heard that Canada gives out As down into the 80% range but I thought I was being fed a line
Yeah 0- 49% is an F 50-59 is a D 60-69 is a C 70-79 is a B 80-89 is an A 90-100 is an A+
It means that 10-20% of exams and assignments can be used to really challenge students without unfairly affecting grades of those who meet curriculum expectations.
If I want to get a better sense of lemmy than headlines, that 47% success at summarizing all the posts is good enough and much faster than I can even skim
If I want to code a new program, that 47% is probably pretty solid at structure and boilerplate so good enough. It can save me a lot of time
If I want to summarize the statuses of my entire team, that 47% may be sufficient for a Slack update to keep everyone up to speed but not enough to send to management
If I’m writing my thesis, that 47% is abject failure
If you miss key information the summary is useless.
If the structure of the code is bad then using that boilerplate will harm your ability to maintain the code FOREVER.
There are use cases for it, but it has to be used by someone who understands the task and knows the outcome they’re looking for. It can’t replace any measure of skill just yet, but it behaves as if it can which is hazardous.