Disaggregated Serving
topic on 1 show · 1 statements across 1 episodes
1 statements about Disaggregated Serving, every show
Ultra-fast network interface cards could deliver 100x speedups in disaggregated AI inference
“If you were to somehow be able to, in like this theoretical dreamland, have extremely fast NICs, you could, in theory, spare that HBM and you could just transfer KVCache trans, like directly from one node to another. This would give you like almost a hundred X…”