Klarna considers AI chatbot error rates acceptable if on par with humans
Sebastian Siemiatkowski · Did Klarna Really Automate 700 Jobs With AI? — With Sebastian Siemiatkowski · Jul 17, 2024 · at 15:19
Sebastian Siemiatkowski, CEO of Klarna, explains the benchmark and quality control framework Klarna uses when comparing customer service AI agents to human staff.
0:00 / 0:19exact quote · 19.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So what we simply do is we read a lot of these transcripts on a continuous basis, and we do continuous quality checks to ensure that the error rate is not higher for the AI chatbot than it is for our human agents, and if we see that they are at least on par, Then we think that's an acceptable outcome”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sebastian Siemiatkowski
Klarna's AI customer support agent absorbed the workload of 700 human reps
“Now the difference was when we Got this customer service AI agent to reach a level where it actually served a lot of errands, and on a satisfaction level equal to what human agents many times did, and we took it live, the number of errands that our human agent…”
Klarna CEO: AI agent handles 1.3M monthly errands, replacing 800 workers
“Our agent that we launched about a year ago is still dealing with, I think it is about 1.3 million errands per month, which is the equivalent of previously was done By about 800 people on a monthly basis. And so that is obviously, and it's actually gone up a l…”
Klarna cut outsourced customer service spend by $40 million using AI
“We on average would have like a about 3000, it depends on because you have to remember also like our, just like Amazon, We have much more transactions around Christmas because we're very online, you know, so there will always be variations in these numbers, bu…”
Siemiatkowski: Viral 700-person layoff figure was entirely coincidental to AI gains
“So the layoff of like the comparison to the seven under layoff is actually a misquote by a news magazine online. It's not accurate. It was two years ago when we had to you know, change the amount of investments we were doing, we had to make layoffs and it just…”