Ali Taha
Team Member, Baseten · 1 appearance on the record.
computed by AI from the episodes · how this works → · full disclaimer →
Ali Taha works at Baseten on model inference infrastructure. He focuses on optimizing production APIs through techniques like speculative decoding and cache-aware routing.
21statements → 7claims → 2claims resolved → 3.71/5average certainty → 2.57/5average debate potential → ≈4.5/5argument clarity, estimated →
2 supported 0 partly supported 0 contradicted 5 not checkable as stated how the 7 claims stand · each chip opens the sources
3 predictions · 4 assertions · 5 opinions · 7 insights · 2 disclosures · every statement was checked. The predictions and assertions are the 7 claims: statements the public record can support or contradict. 2 are resolved, and 5 name no date, number or outcome precise enough to check. Everything else (opinions, insights, what ifs, disclosures) can never be settled by the record, so it carries no assessment.
The record, in short
What the tape says about how Ali argues and how the claims held up. Everything they said, and everything said about them, is in the tabs below.
Their most notable supported claim
Retrofitting Grouped Query Attention layers cures inference bottlenecks and maintains acceptance
“So we find it better to, like, okay, we're going to replace this, you know, we're going to replace this layer with a layer from another model that's using, like, GQA, for instance. And then just for the right training, you can get it to have the same acceptanc…”
How they sound: not measured why? →
We measure speaking style by listening to the audio itself, and a fair number needs at least 2,000 words from one person on tape we have measured. There is too little of Ali Taha on measured tape to publish a rate. This says nothing about how they speak.
Everything Ali Taha said on Latent Space that made the record, most notable first. Filter by type, assessment or year in the ledger →
AI ASIC startups are doomed as NVIDIA GPUs become domain-specialized
“How can you look at this trend and then still be bullish on companies that are coming up with Asics for AI.”
GLM-5.2 autonomously wrote and guided production GPU kernels for Baseten's inference engine
“Some of the GPU kernels that were on GLM-Five-two within our inference engine is written by GLM-Five-two. And the trace and the kernels were guided by GLM-Five-two as the driver.”
Quantizing more layers can actually improve model fidelity via error cancellation
“It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in is that you can predict which …”
Optimal inference parallelism cannot be mathematically calculated; it must be auto-tuned
“And with training, it's more of like a math, like you can run the math and see the flops and maximize it. With inference, it's more of like an auto-tuning, like GPU kernel auto-tuning... You shadow the same traffic, like real traffic, and you just see which co…”
Companies developing fused mega kernels rarely run them in production
“Even the companies that have worked at or people that have spoken to who work at companies that do fused mega kernels, they, Very, very often don't end up running those in production because the TRTL and modular kernels that will launch are faster because you …”
NVIDIA's Vera Rubin GPU architecture effectively renders mega kernel research obsolete
“The GPU is, is It's designed in such a way that it basically kills megakernels. You don't need to use megakernels that much anymore. So it seems like that entire research field goes into, like, won't be continued, but yeah.”
Open-source LLMs have reached closed parity, but video models lag far behind
“The difference between the best open source LLM and the best open closed source LLM is very small. Like it used to be six months. I don't think it's six months anymore. I think it's like basically almost unparative. Video models are definitely not, there's a h…”
Long-form AI video generation must switch to autoregressive architectures over diffusion
“Autoregressive video seems to me like that is the bet that the future is going to be making. But there are no good open source autoregressive video models out there today. And that seems to be, if you want to get like an hour movie, if you want to see video mo…”
Ultra-fast network interface cards could deliver 100x speedups in disaggregated AI inference
“If you were to somehow be able to, in like this theoretical dreamland, have extremely fast NICs, you could, in theory, spare that HBM and you could just transfer KVCache trans, like directly from one node to another. This would give you like almost a hundred X…”
Every current autoregressive video model has terrible output quality compared to diffusion
“The downsides is every single
Autoregressive video model is shit.
It's just terrible quality.
If you, I like, it's just, if you put the quality of any opens like one two or two versus any other autoregressive model, you can see like a video generated by one tw…”
Editing isolated facts in MLP weights fails to update multi-hop downstream reasoning
“So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to like, use its knowledge to think and then give me a second answer, or like, should I hir…”
KV cache compaction is the viable path to solve LLM continual learning
“If you're able to sort of make your KV almost infinite, and you're able to compact in such a way that you don't lose any of the knowledge. In that case, you can actually do a continual learning as you can actually solve continual learning. And this, as a, it's…”
Hourly GPU instances are cheaper than pay-per-token for high-volume AI inference
“And more often than not, it's like way cheaper if you're pushing like millions of tokens per hour, if you just pay per hour instead of pay per token.”
Unclosed JSON syntax is the primary cause of model tool hallucinations
“The challenge with tool calling more and more seems to be that the companies want certain tool calling, which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't like close the end of the request in a very…”
Retrofitting Grouped Query Attention layers cures inference bottlenecks and maintains acceptance
“So we find it better to, like, okay, we're going to replace this, you know, we're going to replace this layer with a layer from another model that's using, like, GQA, for instance. And then just for the right training, you can get it to have the same acceptanc…”
NVIDIA or model creators will always publish open quantized checkpoints
“There's always going to be like an open source quantized checkpoint. NVIDIA is going to push one out if no one else does. You usually the providers will have their own spec tech that they've trained as well. You don't need to train your own spec tech. You can …”
Speculative draft models are typically one-sixteenth the target model's size
“Like for Minimax, it's, yeah, yeah. It's like one layer. It's like one 16th of the original model usually.”
TurboQuant is inefficient on high-bandwidth data center GPUs like NVIDIA B200
“TurboQuant would not be like, it would not be used. Like Nvidia made it clear that this is not a good optimization. And we've seen it firsthand where the overhead of doing dequantization, quantization of You know, in the kernel itself, the turbo-quant kernel, …”
Fused GPU kernels cannot eliminate inter-GPU communication in tensor parallelism
“Well, I mean, like a fused kernel can't save you. Like here with the tensor parallelism, you're the half, the matrix is one GPU and the other half is on another. And if I need the entire matrix in order to do like a non-linear operation on the next step, which…”
Network interface card speed is the primary bottleneck in large-scale AI serving
“I think the answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottom, like you want to have larger models. Right now, when you're doing serving at large, you have to transfer KV cache from…”
Baseten terminates or reprocesses generations after four identical repeating tokens
“We have, like, in our endpoint, like, if a model was to output the same exact token, like, four plus times, we just call the generation, we say, like, oh, sorry, this, like, try again, or, like, we will re-process the request. Because we know then, like, if it…”
Appearances (1)
| Episode | Date | Speaking time |
|---|---|---|
| Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha | Aug 3, 2026 | 32m |