In Context Learning
topic on 4 shows · 6 statements across 6 episodes
Startups For the Rest of Us 
Latent Space 
No Priors 
TBPN
6 statements about In Context Learning, every show
Ries: AI in-context learning inevitably produces average work
“What they call in context learning is not learning. It's just token prediction does next token guessing. So what happens is it just, it inevitably convinces itself To do average work, because that's what's in the training data.”
Bissell: Activation Steering and In-Context Learning Are Quantitatively Equivalent
“He actually has a paper that, as well as some, you know, others from the team and elsewhere, that go into the essentially equivalence of activation steering and in-context learning, and how those are from a, he thinks of everything in a cognitive neuroscience …”
Patel: Prompt correction is an order of magnitude worse than human learning
“I think the, this is at least an order of magnitude less efficient and less less capable than the way humans learn.”
Providing agents with RAG factory blueprints yielded zero benchmark score improvement
“When you try and move that into a benchmark setting with already pre-trained models, just using in-context learning, it's just not that helpful. A thousand lines of Python telling you how to make this kind of factory unit, which it may not be directly applicab…”
Goyal: In-context learning outperforms fine-tuning in many large-context cases
“There's a lot of cases now, especially with large context models, where in context learning just beats fine tuning.”
Liu: Fine-tuning medium models can harm their in-context learning ability
“And if you fine-tune a medium-sized-ish model, sometimes it loses the ability to do effective in context learning, because I think the intuition is, it's devoting more, more of its parameter space to, kind of, like, memorizing the training set so it can do bet…”