Long Context Window Models
topic on 1 show · 1 statements across 1 episodes
1 statements about Long Context Window Models, every show
Liu: Long-context recall depends directly on needle-in-haystack training loss
“The skill with which models are able to take advantage of context is always going to be dependent on how that factors into the impact on the training loss, right? So like, If you want long context window models to work well, then you have to have a ton of data…”