open source serving frameworks
1 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Jul 28, 2025 by Varun Mohan · across every show →
Everything said about open source serving frameworks, oldest first
Jul 28, 2025 negative
Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. Bu…”