Meta trained Llama 3 8B and 70B models on 15 trillion tokens
Meta's Generative AI Head · Meta's Generative AI Head: How We Trained Llama 3 · Apr 22, 2024 · at 5:43
Ahmad Al-Dahle, Meta's VP of Generative AI, details the training scale and dataset size used to develop Meta's Llama 3 models.
0:00 / 0:18exact quote · 18.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So if you look at something like the eight billion and seventy billion, They were trained on almost 15 trillion tokens and tokens roughly you can imagine as a word. So roughly like 15 trillion words, which is an incredible outcome.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Meta's Generative AI Head
Ahmad Al-Dahle says Meta AI will be the best free assistant
“And meta AI is going to be the, you know one of the best if not the best assistant that's available for free.”
Meta releases 8-billion and 70-billion parameter Llama 3 models
“We are releasing an updated eight billion parameter model plus a seventy billion parameter model. And these are state of the art.”
Meta leveraged synthetic data during Llama 3 post-training phase
“One of the things that we did with Lama three is in post-training we actually leveraged synthetic data.”
Ahmad Al-Dahle doubts the AI industry will hit a data wall soon
“I don't think we know yet that, you know, we'll run out of data any, or that there's some like limiting factor here.”
Ahmad Al-Dahle: AI models degrade user experience by over-moralizing refusals
“Some of these models, for example tend to do a lot of moralization or like really take a perspective or a point of view. And we worked on and I'm continuing to work on and innovate on how, how the model responds and how it refuses, which I think is also part o…”