Jul 25, 2024 · 44m · mad
Making AI Work: Fine-Tuning, Inference, Memory | Sharon Zhou, CEO, Lamini
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck interviews Sharon Zhou, CEO of Lamini, exploring the enterprise shift from AI hype to mission-critical deployments. Zhou details Lamini's memory tuning architecture, inference economics, multi-GPU optimization, and the future of continual model learning.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 21.3% of the talking time here. How this is scored →
Matt as informed peer 3.5 Guest teaching 4.1 Guest disagreement 0.1 Matt pushing back 0.1
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 37:45 Critique of 30-call AI agent workflows
Sharon gently rejects the popular developer trend of chaining 30 sequential LLM calls inside agent workflows, pointing out that compound error rates and latency make it impractical for production.
Hardest push from Matt ▶ 16:35 Testing GPU availability claimsMatt pushes Sharon on whether enterprise GPU shortages are actually resolved, asking if major corporate institutions can realistically acquire compute right now.
Biggest teaching moment ▶ 29:10 Explaining parameter-efficient fine-tuning via LoRASharon provides a clear technical breakdown of LoRA, explaining how tuning external parameter matrices yields a 10,000x efficiency gain without adding inference latency.
Matt holds his own ▶ 23:25 Citing Lamini's 52x throughput benchmarkMatt displays deep preparation by quoting Lamini's exact benchmark claim of delivering 52x higher queries-per-second than open-source vLLM.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome Back and Episode Format Overview | 3 | 3 | 0 | 0 | Matt establishes a relaxed interview tone and asks Sharon about current sentiment in the enterprise AI market. Sharon outlines how enterprises are progressing from shallow workflows like email writing to deeper domain-specific problems like insurance field training. | |
| Enterprise Organizational Alignment and Model Evaluation | 3 | 4 | 0 | 0 | Matt inquires about organizational structures and open-source vs proprietary model adoption among large enterprises. Sharon explains that general-purpose LLMs optimize for average web generalization error, making them unfit for tasks requiring precise factual accuracy. | |
| Introduction to Lamini and Memory Tuning Concept | 4 | 3 | 0 | 0 | Matt accurately highlights the market commoditization of inference and asks Sharon to define inference for the audience. Sharon explains Lamini's strategy of offering free inference tokens to convert enterprise customers to their fine-tuning suite. | |
| Mapping the Inference Landscape: Latency, Hardware, and Security | 4 | 4 | 0 | 0 | Matt prompts Sharon to map out the current inference ecosystem and key players. Sharon breaks down the landscape across latency performance, hardware specialization like Groq, and deployment security tiers. | |
| Enterprise GPU Shortage Evolution and Compute Availability | 4 | 4 | 0 | 1 | Matt asks how enterprise inference works amidst reported GPU shortages and asks whether large enterprises can actually get compute. Sharon clarifies that institutional GPU access has eased significantly compared to last year, with A100s and AMD chips widely available. | |
| Hardware Agnosticism, NVIDIA Partnership, and Open Source Foundations | 3 | 3 | 0 | 0 | Matt asks about hardware agnosticism and AMD compatibility in Lamini's architecture. Sharon announces a new partnership with NVIDIA on the spot and explains how open foundation models reduce customer costs. | |
| Technical Definitions: Pre-Training vs. Instruction Tuning vs. Memory Tuning | 4 | 5 | 0 | 0 | Matt asks Sharon to define technical terminology around training pipelines. Sharon provides a clear pedagogical breakdown moving from pre-training to instruction fine-tuning and memory tuning. | |
| Multi-GPU Optimization, Throughput, and Structured Output | 5 | 5 | 0 | 0 | Matt cites specific performance claims regarding Lamini's 52x higher queries-per-second relative to vLLM. Sharon explains multi-GPU hardware utilization bottlenecks and how re-engineering decoders guarantees structured JSON output. | |
| Explaining Key AI Concepts: LoRA and Mixture of Experts | 4 | 6 | 0 | 0 | Matt asks Sharon for accessible definitions of key technical concepts like LoRA and Mixture of Experts. Sharon delivers educational explanations on parameter-efficient fine-tuning and routing across expert adapters. | |
| Enterprise Case Study: 95% Accuracy in Text-to-SQL | 4 | 5 | 0 | 0 | Matt asks if memory tuning is deployed in production and connects case study findings to industry-wide hallucination issues. Sharon explains how memory tuning boosted text-to-SQL accuracy from 50% to 95% for a Fortune 100 enterprise by driving loss to zero on specific schemas. | |
| AI Agents: Architecture, Multi-Call Pitfalls, and Specialization | 3 | 6 | 2 | 0 | Matt introduces the topic of AI agents in enterprise setups. Sharon offers a mild critique of current market trends, warning against agent designs that chain dozens of sequential LLM calls due to latency and error compounding. | |
| What's Next for Lamini, Continual Fine-Tuning, and AI Research | 5 | 5 | 0 | 0 | Matt brings up the core debate between symbolic reasoning and pure compute/data scaling. Sharon offers a researcher's perspective, emphasizing architectural compute efficiency and dynamic model plasticity over symbolic approaches. | |
| Conclusion and Call to Action | 0 | 0 | 0 | 0 | Matt provides a brief, standard monologue outro thanking listeners and encouraging subscriptions. As a solo monologue wrap-up, all host interaction scores remain zero. |
Statements from this episode (18)
Zhou: Generative AI is notoriously hard to evaluate without domain experts
“Generative AI is famously, famously very hard to evaluate. We like have no idea what's good, better, best, unless someone who's an expert in understanding that That use case can tell you that, right?”
Coding AI advanced faster because developers evaluate their own outputs
“And I think that's why, realistically, we've seen many more, like, code agents and applications around code, maybe Texas SQL. We've seen that a lot because the developer Can also evaluate the outputs, right? They are the one in the same person.”
General-purpose LLMs optimize for average internet error, perfect at nothing
“These models, when they're general they're optimizing for what's known as generalization error, or the average error across all examples it sees on the internet. And as a result, it's pretty good at everything, but it's perfect at nothing.”
Memory tuning eliminates hallucinations and enables near-perfect task performance
“Been able with memory tuning, which is what I've been working on to remove those hallucinations, to remove that and actually get these models from, you know, not necessarily being general for everything. And instead of being pretty good at everything, but perf…”
Zhou: Lamini deploys enterprise LLMs on-premise in air-gapped environments
“We're an integrated inference and fine tuning platform for enterprises to be able to run factual LLMs. So essentially LLMs that don't hallucinate on their proprietary data within their secure walls. So we can deploy on premise air gapped, no internet sites. So…”
The AI inference market for startups is a race to the bottom
“My view of inference today, you know, from a market perspective and, you know, startup market perspective is that it's a race to the bottom today for cost. I don't think that's a controversial statement at all. I think people know that it's getting priced lowe…”
Lower LLM latency requires specialized hardware over GPU software platforms
“Unfortunately, I think what people don't realize is that the way to get better latency, like significantly better latency, is actually in the hardware. And that's why we see Grok, G-R-O-Q be able to exceed all these GPU-based inference platforms significantly,…”
The enterprise GPU shortage has eased at the company level
“Today, actually, I'm seeing the GPU shortage go away at the level, at the company level, meaning companies are able to procure enough compute enough is a strong word, but they're able to procure compute at some level to work with, to fine tune and run heavy in…”
In mid-2023, multi-billion dollar companies could not obtain AWS GPU nodes
“Last year was, at this time, was absolutely insane. That's why we threw up our own cloud, because there was just like, large companies with multi-billion revenue numbers could not get a node from AWS, despite their accounts being tens of millions or hundreds o…”
NVIDIA A100s and AMD MI300s are readily available, but H100s remain scarce
“A 100 in particular are pretty available. Obviously the AMD chips that we also agnostically work with the MI 300 and MI two fifties, those are available. H 100 still kind of. A little bit harder to get, but you can get started very easily with any of those oth…”
Memory tuning embeds enterprise data to enable near-deterministic factual recall
“To be able to embed facts of your data into the model, so memory tune the model so that it can recall those facts almost deterministically within its probabilistic context.”
Re-engineering LLM decoders can guarantee absolute schema accuracy for structured outputs
“So that's something else we offer through our inference service to actually make it a hundred percent by re-engineering the decoder of any LLM.”
Future AI models will deliver 100B parameter intelligence at 1B speeds
“I even think there's a future where these models can be a hundred billion parameters, but at, you know, have that intelligence of a hundred billion parameters, but then have the speed, latency, and cost of something that's still one billion or seven billion pa…”
Fine-tuning a GPT-3 class model with LoRA provides a 10,000x efficiency boost
“I think for something like a GPD three level model, it's a 10,000 X speed up in efficiency while losing nearly not much at all in accuracy.”
Combining MoE and LoRA will eliminate big versus small model trade-offs
“And I do think that's the future so we can get something that is incredibly smart, incredibly huge, but with the latency cost and speed of something, something tiny. So no more big model versus small model paradigm. It's potentially one in the same.”
Adding sequential LLM calls or filters to catch errors fails in production
“It's both of those things, and I think people are addressing error today by adding more calls to the model of filtering. Out the requests. And I think I don't think that'll work for serious production use cases.”