Open Source Serving Frameworks
topic on 1 show · 1 statements across 1 episodes
1 statements about Open Source Serving Frameworks, every show
Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. Bu…”