synthetic data

“One of the things that we did with Lama three is in post-training we actually leveraged synthetic data.”

“It will make training more expensive because instead of just doing one backward pass, you now potentially have to do many forward passes because at each forward pass, you're going to come up with some output. Then the model has to decide which of those outputs…”

“More and more synthetic data is becoming a huge chunk of the data that we train on.”

“Synthetic data Probably doesn't get us out of that, that issue. I actually, I don't know if synthetic data outside of easily verifiable domains like math, it's hard to use synthetic data to drive outcomes.”

“One of the things that we've seen over the past few past year in particular is that synthetic data has not worked as well as I think everybody had hoped. You know, pure synthetic data, just using data generated from the models to try to train future models, th…”

“Using humans to train models is just so expensive, right? So then there's the magic of sort of reinforcement learning and other synthetic data technologies, right? Where the model is helping teach the model, right? So you have many models in, in, in a sort of,…”

“I don't think in the next few years it's gonna be the big differentiator. I think that more synthetic data, more human feedback and high quality data is gonna be the differentiator.”

“We are not data constrained right now, we're generating vast amounts of high quality synthetic data, which is proving to be useful.”