Text diffusion models will not replace autoregressive Transformers at state-of-the-art
Sebastian Raschka · State of LLMs 2026: RLVR, GRPO, Inference Scaling — Sebastian Raschka · Jan 29, 2026 · at 12:57
Sebastian Raschka, AI researcher, evaluates whether text diffusion approaches will supersede auto-regressive Transformer architectures in frontier LLMs.
0:00 / 0:10exact quote · 10.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So it is a interesting direction to go into these diffusion, diffusion models as alternative to the auto regressive transformers, but it is not I would say the replacement at the state of the art.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Sebastian Raschka
Future LLMs will prioritize architectural efficiency over larger model sizes
“I wouldn't expect bigger architectures. I would expect a more efficient architectures tweaks getting, The same modeling performance for less compute”
Pre-training is no longer where the low-hanging AI gains lie
“Pre-training is not dead, but pre-training is boring. So it's not where the low hanging fruit is anymore.”
RLVR unlocks pre-training knowledge rather than teaching LLMs new math
“The knowledge is already there in the pre-training, and this just unlocks it. It's just like a step that maybe shows the model how to use its own knowledge, basically.”
Process Reward Models will eventually become standard in LLM post-training
“I think it is promising and we will see it working at some point. I think it's just like right now it's still Tricky to make it work, but I am quite sure we'll see it as part of the standard repertoire at some point.”