The Ledger, every show

“So it is a interesting direction to go into these diffusion, diffusion models as alternative to the auto regressive transformers, but it is not I would say the replacement at the state of the art.”

“I wouldn't expect bigger architectures. I would expect a more efficient architectures tweaks getting, The same modeling performance for less compute”

“Pre-training is not dead, but pre-training is boring. So it's not where the low hanging fruit is anymore.”

“The knowledge is already there in the pre-training, and this just unlocks it. It's just like a step that maybe shows the model how to use its own knowledge, basically.”

“I think it is promising and we will see it working at some point. I think it's just like right now it's still Tricky to make it work, but I am quite sure we'll see it as part of the standard repertoire at some point.”

“That's where you make the bigger gains rather than scaling the model size. I think that's one of those things where you will see more progress coming from.”

“I know for a fact that big companies are training now LLMs in-house. Really, like, big companies who have the financial means to train chat to be like model are hiring people who train LLMs.”

“I would say right now, yes, because it's still the state of the art. So there is nothing really better in terms of state of the art performance, getting better quality results.”

“And you can actually, in fact, Take a GPT one or two model and with a few, I mean, few lines of code almost, you can transform it into the latest let's say deep seek version, 3.2 architecture. It's not like a big leap. It's still the same as scaffold.”

“If you have a business problem, you are maybe manufacturing something, maybe you can start with a generalist model, but then once you know exactly what the task is and you want to hone in on it, maybe it makes sense to replace that expensive thing by something…”

“I think one company will launch a big diffusion model this year.”

“I took the Quinn three model as part of my book, the reasoning from scratch book. And I trained it just for 50 steps with RLVR, and it goes from 15%, so one five percent accuracy on math 500 to 50% on math 500.”

“Deep seek version three, they had like a five million dollar price tag on that given the, I think two dollars per GPU, they assume whether that's a correct assumption or not it's a different question, but if you compare it relative to the cost of R one, I thin…”

“Maybe it's not the one shot problem anymore where it's not really answering knowledge question. That's not really solving math problems in, in one iteration of the benchmark. It is maybe more like the agentic cycle, like where you have like a more like a objec…”

“It rewards the style more, more than the correctness because there is no correctness check.”

“Like if you use or compare ChatGPT, Gemini Claude, Grock. I think they are all pretty much on the same level. Like, and I think that's because they're trying to do everything. Like the generalist models for a general person to do a lot of things. I mean, Claud…”

“If you have an LLM that self improves or like an agent that does something fails and learns, I don't think anything like that is feasible this year.”

“For blog writing on book writing, not so much because honestly, I, for fun, I tried it out. It's just, it generates okay text, but it's, I don't know, it does not I can ask it to generate text like me, but it's almost like, then I don't like it and I end up ed…”

“And I do think that is something that is maybe more expensive to do, but it is also something that might push the state of the art a little bit.”

“Vanilla GRP or the original algorithm, it is pretty flaky. Like where it is, you have to babysit it. Over the course of the year, many people had these tips and tricks where some people were saying, remove the KL divergence term. Like if you just drop it for m…”

“It's similar to, you know, optimizers with Adam. So Adam is, I mean, right now there's Adam W. There was SGD and all the other RMS prop and how they were called. And they kind of all converge to Adam W by adding more and more tricks. And I think that's the sam…”

“The only thing is we are running out of is really benchmarks. So the improvement on benchmarks, it's kind of like harder to measure.”

“And that is very, very powerful because I think this is one of the Ways you can mitigate not totally mitigate, but let's say reduce hallucinations because then the LM suddenly doesn't have to remember everything anymore.”

“And also you can actually go to the GPT OSS release block, and they did have benchmarks to show how the performance on the benchmarks is with the same model with tool called enabled and disabled. And you can actually see there is, I mean, it's not like two tim…”

“Hierarchical reasoning model, it became like popular because it performed relatively well on that benchmark compared to very expensive models like Gemini, Chachupiti, and so forth. And it is a transformer architecture.”