Sebastian Raschka
On 1 show, 1 appearance. The Shows tab opens the full record on each.
1shows
1appearances
25statements
7resolved
6supported
0contradicted
86%fully supported
Everything Sebastian Raschka said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Text diffusion models will not replace autoregressive Transformers at state-of-the-art
“So it is a interesting direction to go into these diffusion, diffusion models as alternative to the auto regressive transformers, but it is not I would say the replacement at the state of the art.”
Future LLMs will prioritize architectural efficiency over larger model sizes
“I wouldn't expect bigger architectures. I would expect a more efficient architectures tweaks getting, The same modeling performance for less compute”
Pre-training is no longer where the low-hanging AI gains lie
“Pre-training is not dead, but pre-training is boring. So it's not where the low hanging fruit is anymore.”
RLVR unlocks pre-training knowledge rather than teaching LLMs new math
“The knowledge is already there in the pre-training, and this just unlocks it. It's just like a step that maybe shows the model how to use its own knowledge, basically.”
Process Reward Models will eventually become standard in LLM post-training
“I think it is promising and we will see it working at some point. I think it's just like right now it's still Tricky to make it work, but I am quite sure we'll see it as part of the standard repertoire at some point.”
Bigger LLM gains will come from multi-model process refinement, not scaling
“That's where you make the bigger gains rather than scaling the model size. I think that's one of those things where you will see more progress coming from.”
Large enterprises are secretly hiring teams to train ChatGPT-scale LLMs in-house
“I know for a fact that big companies are training now LLMs in-house. Really, like, big companies who have the financial means to train chat to be like model are hiring people who train LLMs.”
Transformer remains state of the art for LLM performance
“I would say right now, yes, because it's still the state of the art. So there is nothing really better in terms of state of the art performance, getting better quality results.”
DeepSeek's architecture is still built on a GPT-2 scaffold
“And you can actually, in fact, Take a GPT one or two model and with a few, I mean, few lines of code almost, you can transform it into the latest let's say deep seek version, 3.2 architecture. It's not like a big leap. It's still the same as scaffold.”
Enterprises should transition from generalist LLMs to cheaper task-specific models
“If you have a business problem, you are maybe manufacturing something, maybe you can start with a generalist model, but then once you know exactly what the task is and you want to hone in on it, maybe it makes sense to replace that expensive thing by something…”
Major AI firm will launch a frontier text diffusion model in 2026
“I think one company will launch a big diffusion model this year.”
50 RLVR steps boosted Qwen 3 MATH-500 score from 15% to 50%
“I took the Quinn three model as part of my book, the reasoning from scratch book. And I trained it just for 50 steps with RLVR, and it goes from 15%, so one five percent accuracy on math 500 to 50% on math 500.”
DeepSeek-R1 cost roughly $300,000 to train, 10x cheaper than DeepSeek-V3
“Deep seek version three, they had like a five million dollar price tag on that given the, I think two dollars per GPU, they assume whether that's a correct assumption or not it's a different question, but if you compare it relative to the cost of R one, I thin…”
AI evaluation will shift from single-shot answers to multi-step agentic execution
“Maybe it's not the one shot problem anymore where it's not really answering knowledge question. That's not really solving math problems in, in one iteration of the benchmark. It is maybe more like the agentic cycle, like where you have like a more like a objec…”
Current AI model leaderboards reward response style over actual factual correctness
“It rewards the style more, more than the correctness because there is no correctness check.”
Leading generalist models like ChatGPT, Gemini, and Claude show functional parity
“Like if you use or compare ChatGPT, Gemini Claude, Grock. I think they are all pretty much on the same level. Like, and I think that's because they're trying to do everything. Like the generalist models for a general person to do a lot of things. I mean, Claud…”
Self-improving AI and continual learning agents will not be feasible in 2026
“If you have an LLM that self improves or like an agent that does something fails and learns, I don't think anything like that is feasible this year.”
Raschka does not use LLMs to write his books or blog posts
“For blog writing on book writing, not so much because honestly, I, for fun, I tried it out. It's just, it generates okay text, but it's, I don't know, it does not I can ask it to generate text like me, but it's almost like, then I don't like it and I end up ed…”
Predicting internal states might push state-of-the-art for code LLMs
“And I do think that is something that is maybe more expensive to do, but it is also something that might push the state of the art a little bit.”
Original GRPO algorithm is flaky but stabilizes with practical engineering tricks
“Vanilla GRP or the original algorithm, it is pretty flaky. Like where it is, you have to babysit it. Over the course of the year, many people had these tips and tricks where some people were saying, remove the KL divergence term. Like if you just drop it for m…”
RLVR and GRPO will stabilize into canonical algorithms similar to AdamW
“It's similar to, you know, optimizers with Adam. So Adam is, I mean, right now there's Adam W. There was SGD and all the other RMS prop and how they were called. And they kind of all converge to Adam W by adding more and more tricks. And I think that's the sam…”
The AI industry is rapidly running out of challenging evaluation benchmarks
“The only thing is we are running out of is really benchmarks. So the improvement on benchmarks, it's kind of like harder to measure.”
Tool calling reduces LLM hallucinations by outsourcing memory retrieval tasks
“And that is very, very powerful because I think this is one of the
Ways you can mitigate not totally mitigate, but let's say reduce hallucinations because then the LM suddenly doesn't have to remember everything anymore.”
GPT OSS benchmarks demonstrate 1.2x capability jump when tool calling is enabled
“And also you can actually go to the GPT OSS release block, and they did have benchmarks to show how the performance on the benchmarks is with the same model with tool called enabled and disabled. And you can actually see there is, I mean, it's not like two tim…”
One line per show, most statements first. The link opens Sebastian's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 MADLEDGER | 1 | 25 | 86% 6/7 | full record on the MAD Podcast → |