AI evaluation will shift from single-shot answers to multi-step agentic execution
Sebastian Raschka · State of LLMs 2026: RLVR, GRPO, Inference Scaling — Sebastian Raschka · Jan 29, 2026 · at 38:04
AI researcher Sebastian Raschka explains on The MAD Podcast how AI benchmarks will evolve beyond single-turn evaluation metrics.
0:00 / 0:34exact quote · 34.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Maybe it's not the one shot problem anymore where it's not really answering knowledge question. That's not really solving math problems in, in one iteration of the benchmark. It is maybe more like the agentic cycle, like where you have like a more like a objective that is not, let's say, Answer the question, but more like design something, blah, blah, blah. And then it goes off and how long it can, or how long it needs or how long it can run until the problem is solved. And I think it's maybe more towards that, how we measure progress rather than whether we get 90 or 95% or 97% on a benchmark.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sebastian Raschka
Text diffusion models will not replace autoregressive Transformers at state-of-the-art
“So it is a interesting direction to go into these diffusion, diffusion models as alternative to the auto regressive transformers, but it is not I would say the replacement at the state of the art.”
Future LLMs will prioritize architectural efficiency over larger model sizes
“I wouldn't expect bigger architectures. I would expect a more efficient architectures tweaks getting, The same modeling performance for less compute”
Pre-training is no longer where the low-hanging AI gains lie
“Pre-training is not dead, but pre-training is boring. So it's not where the low hanging fruit is anymore.”
RLVR unlocks pre-training knowledge rather than teaching LLMs new math
“The knowledge is already there in the pre-training, and this just unlocks it. It's just like a step that maybe shows the model how to use its own knowledge, basically.”