Latency
topic on 5 shows · 5 statements across 5 episodes
the Y Combinator Startup Podcast 
Latent Space 
No Priors 
Invest Like the Best 
Big Technology
5 statements about Latency, every show
Jeff Dean: Low latency is critical as AI shifts to complex multi-token tasks
“Latency is actually a pretty important characteristic for these models, because we're gonna want Models to do much more complicated things that are going to involve, you know, generating many more tokens from when you ask the model to do something until it act…”
Amon: Real-time contextual AI cannot run entirely in the cloud
“You cannot do everything on the cloud because of also latency. It is not going to be useful for you. If I go back to when you asked me to describe the experience, if you and I are walking together in the street, and I'm gonna say, hey, who's this person? And y…”
Zhang: Deploying smaller fine-tuned models in AI agents improves performance and latency
“You know how your agent is structured, you know the places where you need models to run, and you can take, you know, smaller fine-tuned models, and that improves the entire system, both in terms of performance, but also latency, and so on.”
Graveley: Every 10ms of latency drops Copilot completions by 1%
“Every 10 milliseconds is one percent fewer completions that people would do.”
Voice AI latency of 200ms versus 100ms is clearly perceptible to users
“That the difference between 50 or a hundred milliseconds of latency and 200 milliseconds of latency is actually quite perceptible. And it really anything we can do to bring that down actually affects user experience quite a bit.”