tail latency

“Running production grade inference is a hard and fair problem. Even if you subtract out the auto scaling, it's controlling things like tail latency and making sure every request is delivered at least once and whatnot.”