synthetic data

“In the instance of self-driving cars, It could make a lot of sense, and I think it can get you 99.9% of the way there, but the question is, how fully reflective is the underlying data set of the real world, and in the instance where it's not 100% fully reflect…”

“The hard part of driving a car autonomously is the edge cases that live beyond 99.99, and I think those ones are just hard to imagine, let alone generate synthetically.”

“It hasn't really worked in practice in most situations, particularly when it comes to visual data or even textual data. It really hasn't worked, and a lot of the reasons why it's sort of limited in terms of its overall impact is that at the end of the day, whe…”

“Now the critical path for model improvement is, is is shifting over to this to this sort of broader sort of simulation slash synthetic data slash like RL loop sort of path. I think it's just a natural consequence of the fact that it's so expensive to just keep…”

“The current LLM API market is dominated by synthetic data.”

“If you have something that can determine an oracle of truth that can help say, this is better and this is worse, or this is correct and this is wrong, that's when you can actually use synthetic data.”

“It's like, well, actually, in fact, we can create synthetic data, and there's a ton of data that's out there that's not part of the standard internet training corpus. So, the scale game is still playing.”

“You could have an LLM generate synthetic data, and when it generates the synthetic data, the data is better. You then train on that synthetic data.”

“Almost all synthetic.”

“A lot of them tell us that even a thousand or a couple of thousand pieces of really high quality human data that we generated for them, it's actually been worth more than ten million pieces of synthetic data.”

“Synthetic data, it's made models good at synthetic problems, not, not real ones.”

“So in some domains, if you think like images, languages, like LLMs talking to each other at some point, you definitely get the degradation and that degradation is due to essentially like a loss of diversity of your data.”

“Look, I think the biggest one is just the view that synthetic data will take over, and you just will not need human feedback.”