Inference Engine
topic on 2 shows · 3 statements across 3 episodes
3 statements about Inference Engine, every show
Goel: Cartesia Built Custom Inference Engine for Real-Time Models
“So as an example, we've built our own inference engine. It's pretty important because we think that these interactive real-time models are going to need to run in a pretty different way. So having the ability to have your own engine that is not designed for LL…”
Speculative decoding creates hardware resource contention and engine orchestration complexity
“The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on. There is a orchestration and resource competition problem inherent in that. And tha…”
Ermon: Inception Labs built proprietary engine for production inference traffic
“So just like you would normally serve an LLM using a VLLM or SGLang or a Tensor or TLLM, we have built our own inference engine. And so we are supporting production traffic already with our own inference engine. We support continuous batch and quantization.”