Hugging Face

“Hugging Face's libraries are actually very inefficient, like incredibly inefficient for inference. You get like, 15% MBU on, on, on, on some configurations, like eight, eight, eight, eight, eight, eight, 100, and LLAMA-seventy-beat, you get like, 15%, which is…”

“Truthful QA is a garbage benchmark, like, you, like, some of the models that are very high on there, if you use it for five seconds, you're like, this is garbage, right?”

“The open LM leaderboard is, like, probably, like, the, a big, like, Problem, to be honest.”

“The hugging face library in peft doesn't really work in practice unless you use it with other things. And there's a lot of coupling in the hugging face ecosystem where, like, none of it works separately. You have to use it all together, which I don't love.”

“Let's also look at the implementation of Mixtro eight by seven on Hagen-Phys transformer. You will soon notice the, in the expert operation there, You would iterate over all of the experts and compute each of the gem operations one by one. We found that this i…”

“I've been a little bit more doubtful on Lama stack. I think you've been more positive. Basically, it's just like the meta version of whatever HuggingFace offers, you know, or TensorRT, or BLM, or whatever the open source opportunity is. But like, to me, it's n…”

“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”

“So it's a series of three models, which are the best in class in each model size. For example, our 1.7 B model outperforms Lama one B and also .2.”

“For example, smaller than one was trained only on one trillion tokens, but this model is trained on 11 trillion tokens. And we saw that the performance kept improving. The models didn't really plateau me training. Which I think is really interesting. It shows …”

“For example, here we measured like these words ratio in different dumps of common crawl, and we can see that like the ratio really increased after chat GPT's release.”

“And then we run this classifier on all of fine web, which is a 15 trillion tokens data set. And then we only keep the pages that have like a score that's higher than three. So for example, in our case, we went from 15 trillion tokens to just 1.5 trillion token…”

“The key ingredient to getting a good data set that is synthetic is trying as much as possible to keep it diverse, because if you just throw the same prompts as your model, like generate, like, a textbook about linear algebra, and even if you change the tempera…”

“And as you can see here FineWebEDU outperforms all the other public web datasets by a larger margin on a couple of benchmarks.”

“Today you can train an LLM with like an entirely synthetic pipeline. For example, you can use our Cosmopedia data sets and you can train a one B model on like a hundred and fifty billion tokens. Those are a hundred percent synthetic, and those are also of good…”

“The main file in it, the agents.py file that we have at the core of the library is under the 1000 lines of code.”

“So I think if we solve Gaia, that's like 90% score. That means mostly we double productivity of every task done in front of a computer. And if you take the trend line of the scores so far this should be crossed in 2026 or something.”

“Looking forward to this, and for instance we're going to fine-tune R-One on agentic stuff, so we'll, we'll get really good powerhouse models soon.”

“That's the next step for us at Hugging Face. We're going to push really hard on building GUI agents, so basically agents that can use any GUI.”

“It was a chat app for teenagers. A lot of people don't know that Hugging Face was like, Hey friend, how was school? Let's trade selfies. But then you know, they built the Transformers library, I believe to help them make their chat app better. And then they op…”

“The way HuggingFace, who built the OpenROne project is using us is during like the reinforcement learn, code gen reinforcement learning step where the ROne model the OpenROne model has a training step where they give it a Code problem, and the model needs to g…”

“I think last I checked, over 300,000 monthly downloads on PyPy. More GitHub stars than Jupyter Notebook for whatever that's worth. And we're used at companies like OpenAI, Hugging Face, Cloudflare, BlackRock, universities like Stanford and Berkeley.”

“The academic community had been using this one data set since like all the way back in the hugging face models of like Zephyr beta is when this ultra feedback data set got popular. And still a year later is like this state of the art data set for open preferen…”

“So rather than having like, we don't have a compiler whatsoever. There's no compiler. There's no translator, no tooling that's involved in actually taking those and getting that to, you know, for your common, you know, Huggy Face Transform models to be able to…”

“We also have this team working on pre-training and training models such as small LM. And we are basically a very small team of We have 30 people in average”

“For example, we tried we are training MOE currently at TargetFace. I mean, we'll train soon. We start the training soon. And we tried with Megatron and we benchmarked, like, for example, the Mistral architecture with the Queen's three this one.”

“We are basically introducing higher quality data like I don't know where it is, but if you look at the different phase, we are basically adding more code and math through the train and less web data. But we're only doing it like one point trillion token.”

“And this is basically until like 6.5 trillion of tokens, which is a lot, to be honest. Until this amount of token, the MMLU in the QA format, meaning that the model have to select which answer is, the model have to output, for example, the right answer is A, o…”

“We want to train our MOE because it's fun and everyone is doing that. And also I think there is a lot of different direction. And it's always good in terms of science. To, because basically the coin tree or even deep seek, they don't release, release all the a…”

“Yeah, I think even I, to be honest, I'm mainly using a closed model because it's easier, basically. It's like the same interface when I'm like, for example, researching very deep on papers.”

“I think that, you know, honestly, SmallLM three, one of the single best open source AI research papers I've read, I think probably in like a year, maybe two years.”

“I do think the next generation of fine tuners will not be, I mean, will be people that are not coding at all, right? Like one year ago, we had to write like our own Colab with Transformers or Oncelot or whichever library of your choice. I do think as we like k…”

“One big part of my job a couple years ago was for any arbitrary model that came out on Hugging Face, writing a config for it and kind of getting it up and running, and now the get it up and running config is, is one-shot-able and so, you know, I don't have to …”