Meta leveraged synthetic data during Llama 3 post-training phase
Meta's Generative AI Head · Meta's Generative AI Head: How We Trained Llama 3 · Apr 22, 2024 · at 11:58
Ahmad Al-Dahle, Meta's VP of Generative AI, speaks to Alex Kantrowitz about Llama 3's training process amidst industry concerns over data scarcity.
0:00 / 0:05exact quote · 5.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“One of the things that we did with Lama three is in post-training we actually leveraged synthetic data.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Meta's Generative AI Head
Ahmad Al-Dahle says Meta AI will be the best free assistant
“And meta AI is going to be the, you know one of the best if not the best assistant that's available for free.”
Meta releases 8-billion and 70-billion parameter Llama 3 models
“We are releasing an updated eight billion parameter model plus a seventy billion parameter model. And these are state of the art.”
Ahmad Al-Dahle doubts the AI industry will hit a data wall soon
“I don't think we know yet that, you know, we'll run out of data any, or that there's some like limiting factor here.”
Ahmad Al-Dahle: AI models degrade user experience by over-moralizing refusals
“Some of these models, for example tend to do a lot of moralization or like really take a perspective or a point of view. And we worked on and I'm continuing to work on and innovate on how, how the model responds and how it refuses, which I think is also part o…”