AI2

“It was the main competitor to Hugging Face Transformers. And they ultimately outcompeted AI two as the thing that people use for that because they had very different model and amount of support.”

“There's like a pool of about 10 trillion tokens from which we have like an algorithm also fully open source. To like sample about six trillion tokens that we use during training.”

“Majority of models that get release I think the best term to describe them is open weights. Your Quinn, your Gemma, your Lama you know, Kimi it's what gets release is a set of weights that correspond either to the final state of model, that's the most common, …”

“We had a bunch of cloud credits and I, they were running out and we're behind and I just generated like as many completions as possible. So it was like a few billion completions from deep seek over the weekend.”

“I would say in post training where The best models that don't start with Quinn three and we're like reasonable to say that they are comparable to Quinn three, like on some benchmarks would beat them on some benchmarks. They're way ahead.”

“So in our case, we took a mix of existing data sets like Open Thoughts three and modified it, which is from Bespoke AI labs, a startup. And then we also generated a whole bunch of new data. So we ended up using a mix of teachers from like Deep Seek R one, oh f…”

“Our initial pool was closer to 300 trillion tokens. You shrink it down till you reach your target number, and hopefully as you shrink, you only keep the best part of this.”

“We spent a long time to try to replicate what we thought was close to Lama three post training with multiple stages and optimizers, which is the project that like came up with the name reinforcement learning with verifiable rewards with a bunch of people.”