The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 32 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Ben Allal: Recent web dumps improve model benchmarks despite synthetic data
“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”
Bakouch: DeepSeek and Qwen do not release all their ablation data
“We want to train our MOE because it's fun and everyone is doing that. And also I think there is a lot of different direction. And it's always good in terms of science. To, because basically the coin tree or even deep seek, they don't release, release all the a…”
Bakouch primarily uses closed AI models for deep research out of convenience
“Yeah, I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. It's like the same interface when I'm like, for example, researching very deep on papers.”
Sohmers: Positron AI requires zero compilers to run Hugging Face models
“So rather than having like, we don't have a compiler whatsoever. There's no compiler. There's no translator, no tooling that's involved in actually taking those and getting that to, you know, for your common, you know, Huggy Face Transform models to be able to…”
Roucher: AI agents will reach a 90% GAIA score by 2026
“So I think if we solve Gaia, that's like 90% score. That means mostly we double productivity of every task done in front of a computer. And if you take the trend line of the scores so far this should be crossed in 2026 or something.”
Ben Allal: Hugging Face SmolLM2-1.7B outperforms Llama 3.2 models
“So it's a series of three models, which are the best in class in each model size. For example, our 1.7 B model outperforms Lama one B and also .2.”
Ben Allal: Small models continue improving when trained on 11T tokens
“For example, smaller than one was trained only on one trillion tokens, but this model is trained on 11 trillion tokens. And we saw that the performance kept improving. The models didn't really plateau me training. Which I think is really interesting. It shows …”
Tay: Hugging Face's Open LLM Leaderboard is a major problem
“The open LM leaderboard is, like, probably, like, the, a big, like, Problem, to be honest.”
Patel: Hugging Face libraries achieve only 15% MBU for inference
“Hugging Face's libraries are actually very inefficient, like incredibly inefficient for inference. You get like, 15% MBU on, on, on, on some configurations, like eight, eight, eight, eight, eight, eight, 100, and LLAMA-seventy-beat, you get like, 15%, which is…”
Bakouch: SmolLM 2 scored random on MMLU until 6.5 trillion tokens
“And this is basically until like 6.5 trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA format, meaning that the model have to select which answer is, the model have to output, for example, the right answer is A, o…”
Ben Allal: LLMs can be trained with entirely synthetic pipelines
“Today you can train an LLM with like an entirely synthetic pipeline. For example, you can use our Cosmopedia data sets and you can train a one B model on like a hundred and fifty billion tokens. Those are a hundred percent synthetic, and those are also of good…”
Shawn Wang doubts Meta's Llama Stack will win broad developer adoption
“I've been a little bit more doubtful on Lama stack. I think you've been more positive. Basically, it's just like the meta version of whatever HuggingFace offers, you know, or TensorRT, or BLM, or whatever the open source opportunity is. But like, to me, it's n…”
Jeremy Howard: Hugging Face libraries suffer from excessive coupling
“The hugging face library in peft doesn't really work in practice unless you use it with other things. And there's a lot of coupling in the hugging face ecosystem where, like, none of it works separately. You have to use it all together, which I don't love.”
Patel: TruthfulQA is a garbage benchmark that gets gamed
“Truthful QA is a garbage benchmark, like, you, like, some of the models that are very high on there, if you use it for five seconds, you're like, this is garbage, right?”
Sanseviero: Next generation of AI fine-tuners will not write code
“I do think the next generation of fine tuners will not be, I mean, will be people that are not coding at all, right? Like one year ago, we had to write like our own Colab with Transformers or Oncelot or whichever library of your choice. I do think as we like k…”
Bakouch: Hugging Face plans to train an MoE model soon
“For example, we tried we are training MOE currently at TargetFace. I mean, we'll train soon. We start the training soon. And we tried with Megatron and we benchmarked, like, for example, the Mistral architecture with the Queen's three this one.”
Swix: SmolLM 3 is the best open-source AI paper in two years
“I think that, you know, honestly, SmallLM three, one of the single best open source AI research papers I've read, I think probably in like a year, maybe two years.”
Lambert: Academia relied on UltraFeedback for open preference tuning for a year
“The academic community had been using this one data set since like all the way back in the hugging face models of like Zephyr beta is when this ultra feedback data set got popular. And still a year later is like this state of the art data set for open preferen…”
Mlejnsky: Hugging Face uses E2B sandboxes during Open-R1 RL training
“The way HuggingFace, who built the OpenROne project is using us is during like the reinforcement learn, code gen reinforcement learning step where the ROne model the OpenROne model has a training step where they give it a Code problem, and the model needs to g…”
Ben Allal: Prompt diversity is essential for scaling synthetic training data
“The key ingredient to getting a good data set that is synthetic is trying as much as possible to keep it diverse, because if you just throw the same prompts as your model, like generate, like, a textbook about linear algebra, and even if you change the tempera…”
Ben Allal: FineWeb-Edu Outperforms All Other Public Web Datasets
“And as you can see here FineWebEDU outperforms all the other public web datasets by a larger margin on a couple of benchmarks.”
Ethan He: Hugging Face's sequential GEMM loop for Mixtral is inefficient
“Let's also look at the implementation of Mixtro eight by seven on Hagen-Phys transformer. You will soon notice the, in the expert operation there, You would iterate over all of the experts and compute each of the gem operations one by one. We found that this i…”
Writing deployment configs for arbitrary Hugging Face models is now one-shot-able
“One big part of my job a couple years ago was for any arbitrary model that came out on Hugging Face, writing a config for it and kind of getting it up and running, and now the get it up and running config is, is one-shot-able and so, you know, I don't have to …”
Hugging Face's pre-training research team averages 30 people
“We also have this team working on pre-training and training models such as small LM. And we are basically a very small team of We have 30 people in average”
Bakouch: Hugging Face introduces code and math in 1T+ token phase
“We are basically introducing higher quality data like I don't know where it is, but if you look at the different phase, we are basically adding more code and math through the train and less web data. But we're only doing it like one point trillion token.”
Marimo Surpasses 300K Monthly PyPI Downloads and Jupyter's GitHub Stars
“I think last I checked, over 300,000 monthly downloads on PyPy. More GitHub stars than Jupyter Notebook for whatever that's worth. And we're used at companies like OpenAI, Hugging Face, Cloudflare, BlackRock, universities like Stanford and Berkeley.”
Sutin: Hugging Face started as a teenage chat app before Transformers
“It was a chat app for teenagers. A lot of people don't know that Hugging Face was like, Hey friend, how was school? Let's trade selfies. But then you know, they built the Transformers library, I believe to help them make their chat app better. And then they op…”
Roucher: Hugging Face will fine-tune DeepSeek-R1 for agent workflows
“Looking forward to this, and for instance we're going to fine-tune R-One on agentic stuff, so we'll, we'll get really good powerhouse models soon.”
Roucher says Hugging Face plans to heavily prioritize GUI agents
“That's the next step for us at Hugging Face. We're going to push really hard on building GUI agents, so basically agents that can use any GUI.”
Hugging Face finds LLM proxy words jumped in Common Crawl after ChatGPT
“For example, here we measured like these words ratio in different dumps of common crawl, and we can see that like the ratio really increased after chat GPT's release.”
Hugging Face Filtered 15T Token Dataset Down to 1.5T Educational Tokens
“And then we run this classifier on all of fine web, which is a 15 trillion tokens data set. And then we only keep the pages that have like a score that's higher than three. So for example, in our case, we went from 15 trillion tokens to just 1.5 trillion token…”