I think that study is the wrong framing of the problem for identifying economic returns on AI. We don't need AI to complete tasks perfectly, just to be able to generate a good enough approximation that is easy to review and correct such that an employee has to spend less time correcting AI's errors than they would spend producing the entire output from scratch. So it won't be a drop in replacement for an employee for another 4-10 years, but in the interim, will shift an employee's role from generating a complete solution to primarily reviewing and correcting an LLM-generated solution to get it from that 80-95% level (or whatever the starting point might be prior to 2029) to 100%.

At this point, the vast majority of the work required to make GenAI capable of producing that sufficiently reviewable/correctable content isn't improving model quality, but creating the harnesses, infrastructure, and workflows around the models. Companies aren't seeing returns yet because too many early adopting companies have conceived of AI as a drop in replacement for employees, or at least as a reason to cut staff immediately, without first building out the supporting systems needed to compensate for the inadequacies of the models.

10% failure rate? Wouldn't that be depending on task disastrous? Or possibly expensive?

I think any juniors who keep failing 10% of text based task will eventually get fired... So investing in those that don't fail seems only sensible move as usual.

Yes, it would.

Something I keep in mind is that the “goal” success/failure rate really needs to be appropriate in context. We sure can’t eliminate human error, but for my work a best case 80/20 would mean I’m losing customers and probably getting myself sued. I don’t have a problem “doing things by my own hand” in that case.

Oof. And how much more expensive will the models be to get that 80%?