Chunking
topic on 2 shows · 3 statements across 2 episodes
the Knowledge Project 
Latent Space
3 statements about Chunking, every show
Eugene Yan: Removing document chunking boosted pipeline downstream metrics by 50%
“I was asked to help with a pipeline, and I was able to improve downstream metrics significantly by 20 to 50% by removing chunking.”
Shreya Shankar: Chunking efficacy is task-specific, requiring automated pipeline optimization
“Sometimes it's beneficial to chunk and sometimes you should not chunk. And we have observed this in a number of workloads and the insight that we've gained is that we will never know what it's all task specific and data specific. And we are so further convince…”
Oakley: Skill acquisition requires linking small cognitive chunks into larger patterns
“First, to be aware that there's a way of crystallizing what you're doing that creates a pattern that is a chunk. And you can make a bigger pattern, right? Just like a chain, and you can add more links to the chain.”