LLM Inference
topic on 5 shows · 7 statements across 7 episodes
In Depth 
Latent Space 
the MAD Podcast 
TBPN 
20VC
7 statements about LLM Inference, every show
Pedregal: Audio transcription is Granola's largest cost, exceeding LLM inference
“So the most expensive thing about our business is actually transcription. And historically it's been actually transcription and high quality transcription versus LLM inference.”
Russ d'Sa: LLM inference is now faster than text-to-speech generation
“Now like LLM inference can actually be done in less time than generating speech with TTS.”
D'Sa: LLM inference is now faster than TTS speech generation
“And now like LLM inference can actually be done in less time than generating speech with TTS.”
Lazarte: Now is the best time in over a decade to start a startup
“It's quite clear. It's the best time in over a decade to start a company. Because we have this insane phenomenon of adoption, like adoption of LLM inference.”
LLM prompt processing is compute-bound; next-token generation is memory-bound
“Prompt processing is bottlenecked by computation, and generating next, predicting next token is bottlenecked by memory bandwidth.”
Patel: AI inference will deploy more GPUs than training by 2024
“LLM inference will be bigger than training, or multimodal, whatever, blah, blah, blah inference will be bigger than training, you know, probably next year, in fact at least in terms of GPUs deployed,”