synthetic data

“Fi-one-point-five has never read Wikipedia, for example, so it doesn't know who Tom Cruise is, you know it doesn't know who anybody is, he doesn't know about any movies, it doesn't really know anything about anything, like, because it was never, it's never rea…”

“I feel like most models today, they still use, like, combination of, like, the stack and the pile as, like their training corpus but you can only stretch that so far. At some point, we need more data and I don't know. I think there's still more alpha in, like,…”

“Outside of this, like, where we don't have good symbolic models, like, synthetic data obviously, like, doesn't make any sense. So synthetic data is not a magic wand where it'll work in all cases, in every case, you know, whatever. It's just where we as humans …”

“I think that translation between natural language, English versus code and back and forth, I think is actually actually a really ripe source of synthetic data and Lama three specifically called out that, that they trained on that.”

“Most VLMs are distillations of proprietary closed source models, right? So if you need to generate synthetic data, like most open weight models rely heavily on synthetic data from private models.”

“So I think on the data side, we're approaching the limit and the only data to increase that is synthetic generated data.”

“Synthetic data, it doesn't have to come from a single model. And because we have so many good models now, you could like pull these models together and get like a dataset that's over really high quality and that's diverse and that's covers all your needs.”

“They rewrite some samples from C four into Q and A into Wikipedia, and they find that doing this works better than training just on C four.”

“College textbooks are really good for MLU or middle school textbooks are good for benchmarks like open book, UA and Pico.”

“So what we did is we trained different models on these different dumps, and we then computed their performance on popular like NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest dumps are actually even bett…”

“And I think there's a lot of concerns about model collapse, and I'm going to talk about that later, but we'll see that like, if we use synthetic data properly and we curate it carefully that shouldn't happen.”

“The key ingredient to getting a good data set that is synthetic is trying as much as possible to keep it diverse, because if you just throw the same prompts as your model, like generate, like, a textbook about linear algebra, and even if you change the tempera…”

“So personally, I wouldn't say the web is posted with synthetic data. Maybe it's even making it more rich.”

“The interesting thing in this approach is that you can use a model that is small Because it doesn't, rewriting doesn't require knowledge. It's just rewriting a page into a different style. So the model doesn't need to have like knowledge that is like extensive…”

“Well, I bet people, I mean, they surely will use these for synthetic data. It's just like the marginal gain on synthetic data is always very high.”

“If you filter the data after each point, that's now information injection, and that can break all of this and I think can prevent model collapse.”

“We were actually generating a lot of synthetic data and showing that, hey, you can actually, these synthetic data are actually as useful as real-world data when it comes to multimodal pre-training.”

“Building a video model. You actually need to build a image model first and building, building these two models. The data you need is a hundred percent synthetic pair of language and image or language to video because on the internet, actually the videos Don't …”

“I think everyone is trying to accumulate like a data, which is not a mode. It's just time and time mode. It's all about like, you know, whether you can execute fast enough to make sure that you have like a certain buffer because of say your data set, you know,…”