Philip Kiely
Engineer & Head of AI Education, Baseten. On 1 show, 1 appearance. The Shows tab opens the full record on each.
authorengineeroperator@philipkiely ↗LinkedIn ↗philipkiely.com ↗
Philip Kiely is an early engineer at AI infrastructure company Baseten, where he focuses on the software and hardware stacks required to serve, optimize, and scale production AI models. He is the author of Inference Engineering and Writing for Software Developers, educating engineers on low-latency infrastructure and model-serving runtimes.
1shows
1appearances
28statements
5resolved
2supported
1contradicted
40%fully supported
Everything Philip Kiely said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Quantization, speculative decoding, and prefill-decode disaggregation each yield 2x speedups
“Going from BF-sixteen to NVFP four is, it's not quite a two X, right? It's like, I think it's about like 30, 30 to 40% from 16 to eight, and then another 30 to 40% multiplied from eight to four. So that doesn't quite get you a two X, but like roughly a two X. …”
Leading AI agents will utilize continuous inference-to-training learning loops within two years
“And I think within a few months to a couple of years, like a lot of, Leading agent builders are going to have these loops, like, really up and running in production, where you are doing inference, learning from the inference.”
AI customers migrate from pay-per-token APIs to dedicated GPUs upon finding fit
“I think that we've increasingly seen a lot of demand for the sort of pay per token APIs just because everyone wants to try open models. And then once they find a use case that's really sticky then they move over to Dedicated.”
Open-source inference engines often receive model weights before official launches
“Getting to the point of I can make a token out of this model is not that hard because generally the open source inference engines, your VLMs, SGLangs of the world oftentimes even receive weights ahead of time maintainers do, or the people making the model merg…”
DeepSeek models are the hardest to support due to architectural novelties
“I think that, like, obviously the DeepSeq models tend to be the most challenging as they have, like, the most novel architectural stuff going on model over model.”
Modifying base LLM weights for vision degrades original text performance
“You don't want to mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision.”
Baseten's vision-retrofitted GLM-5.2 scored 56% on MMLU Pro without text degradation
“It's not, you know, it got to a 56% on MMLU Pro, I think, so not, not quite Frontier, but if you're running this model, you haven't suffered any loss on your GLM-Five-II quality.”
Inference optimization is only solved once researchers report mere 1% speedups
“Like, you'll, you'll know that influence is pretty much solved when researchers start publishing about how they got one percent faster at something.”
Software optimizations yield 2x to 4x inference speedups on identical hardware
“Yeah, then you're looking at, like, a two to four X improvement, depending on the inference optimizations.”
NVIDIA Dynamo is a developer toolkit, not an out-of-the-box performance optimizer
“I would think of Dynamo as less of a sort of out of box system and more of a toolkit for building with. So when we talk about doing KV aware routing, when we talk about doing KV out offloading, when we talk about doing PD disaggregation, Dynamo fundamentally i…”
Eagle remains widely used despite newer speculative decoding methods like DeFlash
“The speculation space has moved much faster than everything else. So, yeah, even at the time that I wrote the book Medusa, I very much included as a way for people to understand how the, how, how the space evolved rather than what the most modern technique is,…”
Speculative decoding creates hardware resource contention and engine orchestration complexity
“The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on. There is a orchestration and resource competition problem inherent in that. And tha…”
Tensor parallelism is a poor fit for local multi-device AI setups
“Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink was, you know, They might be forced to do something like pipeline parallelism, which we're never going to do unless we're doing some kind of mu…”
All modern AI models requiring multi-GPU parallelization are Mixture-of-Experts
“Effectively, all models today are MOE models that are, you know, at least all models large enough that you would care to parallelize them across multiple GPUs.”
Pipeline parallelism is only necessary for multi-node AI model inference
“The only reason you would have to do pipeline parallelism which is where you separate, like, different layers, and you put, like, half the layers on one hardware and half on another, is if you are forced to do multi-node inference because a model is bigger tha…”
Most open AI research labs avoid hardware-specific model architecture optimization
“Most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip.”
NVIDIA Rubin will shift inference engineering toward traditional hardware infrastructure challenges
“I think that themes around like KV cache offloading, KV aware routing, and disaggregation are going to be substantially more important in the Rubin era, which means that inference engineering becomes not just a like CUDA kernel problem, but also like a very tr…”
NVIDIA's Rubin is the first GPU architecture designed entirely for modern LLMs
“Ruben's honestly the first chip that was fully built in that world. And so you can see a lot of the understanding of the shape of the workload that this chip's going to be asked to do in the way it's designed.”
Current inference hardware capacity sets the parameter ceiling for new foundation models
“Like, people understand the ceiling that you have in terms of how many parameters of a model you can run, given the sort of latest inference hardware, and that kind of forms a ceiling.”
Future long-form video generation will likely hybridize autoregressive and diffusion techniques
“So if we think about the architecture that's going to get us there to these longer, richer sequences, it's probably, like you said, going to be a mix of the auto aggressive and the diffusion working together to do what each piece is good at.”
Dynamic system adjustments consistently outperform static configurations in AI inference setups
“Any kind of dynamic adjustment is going to beat a static configuration across, you know, your exact config, across your speculator, across that kind of thing.”
Training speculative decoding models requires hidden states from the base model
“Now, with speculators today, you need to train the speculator using the base model itself, because you're getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator.”
Most inference optimizations like KV caching are lossless, unlike quantization
“Most inference optimizations are lossless. KV caching, for example, you are just recomputing or preventing recomputing the same values. Speculation, of course, If a draft token is wrong, it gets rejected. The main lossy optimization is quantization”
Unoptimized GLM-5.2 delivers a baseline 30 to 40 tokens per second
“So let's say you have, as a reasonable baseline, 30 or 40 tokens per second. You can achieve 10 X that. So like on GLM 5.2 if you want to get unquantized perhaps on hoppers even and you're just using an off the shelf inference engine with no particular optimiz…”
One line per show, most statements first. The link opens Philip's full record on that show: the calibration, argument clarity, speaking style and every statement made there.
| Show | Role there | Eps | Statements | Record | |
|---|---|---|---|---|---|
 LATENT SPACELEDGER | Engineer & Head of AI Education, Baseten | 1 | 28 | 40% 2/5 | full record on Latent Space → |