long context window models
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Dec 17, 2023 by Beyang Liu · across every show →
Everything said about long context window models, oldest first
Dec 17, 2023 neutral
Liu: Long-context recall depends directly on needle-in-haystack training loss
“The skill with which models are able to take advantage of context is always going to be dependent on how that factors into the impact on the training loss, right? So like, If you want long context window models to work well, then you have to have a ton of data…”