Jun 9, 2026 · 51m · neon-show
94% CAGR: What the Inference Boom means for your AI costs | Vamshi Ambati
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The Neon Show, host Siddharth Ahluwalia interviews AI researcher and entrepreneur Vamshi Ambati to analyze the explosive 94% CAGR growth of the inference market, shifting token economics, and strategic playbooks for scaling enterprise AI startups.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
Siddhartha as informed peer 4.5 Guest teaching 5.4 Guest disagreement 1.6 Siddhartha pushing back 2.2
speaking balance: gold is Siddhartha, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 11:26 Guest rejects premise that building AI software is cheap
The guest directly challenges the host's premise by arguing that compute access remains highly restricted and reasoning token costs will inflate building expenses.
Hardest push from Siddhartha ▶ 13:38 Host challenges guest on 100x compute cost reductionThe host explicitly pushes back against the guest's thesis of rising token costs by citing industry-wide initiatives aimed at making compute 100x cheaper.
Biggest teaching moment ▶ 19:42 Guest provides definitive neural inference breakdownThe guest delivers a foundational explanation of inference versus training, mapping biological neural maturation to LLM parameter forward-pass computations.
Siddhartha holds their own ▶ 8:36 Host uses Nutanix cloud-to-on-prem market share analogyThe host demonstrates deep domain knowledge by invoking historical on-prem survival rates and Nutanix's creation to test the model layer's true enterprise penetration ceiling.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Siddhartha as informed peer | Guest teaching | Guest disagreement | Siddhartha pushing back | Why |
|---|---|---|---|---|---|---|
| The Three Waves of AI and Unprecedented Innovation Pace | 3 | 6 | 1 | 1 | The host opens with a broad macro question about the AI landscape. The guest provides a comprehensive 20-year retrospective detailing the three distinct waves of AI from symbolic to statistical to neural. | |
| Enterprise AI Adoption, Trust, and SaaS Disruption | 5 | 5 | 2 | 3 | The host utilizes historical context about on-prem versus cloud adoption and mentions Nutanix to query the model layer's market share. The guest acknowledges the host's intuition before explaining why enterprise trust and reliability will determine the pace of diffusion. | |
| The Economics of Token Pricing, Hardware, and Reasoning Costs | 6 | 6 | 3 | 5 | The guest questions the host's assertion that building software has become cheap due to token and compute limits. The host pushes back by pointing out that predictions of rising model costs run counter to industry efforts to make compute 100x cheaper. | |
| The 94% CAGR Inference Market and Defining Inference | 6 | 6 | 1 | 1 | The host cites AWS revenue figures to frame a question on whether inference will outgrow compute spending. The guest delivers market forecast data showing a 94% CAGR and provides a detailed analogy comparing training and inference to human brain development. | |
| Core AI Verticals: Deterministic Code vs. Human Customer Voice | 5 | 5 | 3 | 4 | The guest breaks down the deterministic advantages of coding and expresses skepticism regarding AI automation of human-to-human touchpoints. The host counters this skepticism by citing decacorn valuations achieved by customer support AI platforms like Sierra and Decagon. | |
| Market Trends: Agent Hype vs. Undervalued Voice Modality | 3 | 5 | 2 | 1 | The host asks what is overvalued and undervalued in the current AI market. The guest takes a contrarian stance against near-term agentic hype while highlighting voice modality as an undervalued tool for widespread literacy and adoption. | |
| Origins of Predera: Forward-Deployed Research in Healthcare | 5 | 6 | 1 | 1 | The host explores the transition from a services model to product development. The guest walks through his forward-deployed experience embedded in hospital systems, navigating complex buyer personas and the healthcare 4Ps before expanding across industries. | |
| The Strategic Pivot to LLMOps and Navigating Dual Exits | 4 | 4 | 1 | 2 | The host presses on the financial breakdown and structure of the exits. The guest recounts walking away from a Walmart contract renewal to pivot fully toward LLMOps ahead of two distinct acquisitions. | |
| Services vs. Product Frameworks for AI Startups | 5 | 6 | 1 | 3 | The host questions whether forward-deployed engineering risks turning a startup into a bespoke single-client shop and asks about Palantir's model. The guest shares a strict five-customer rule across distinct verticals required to validate true product repeatability. | |
| Technical Founder to Enterprise Sales: Landing Fortune 500 Deals | 3 | 5 | 1 | 1 | The host asks for parting advice on how a technical founder evolves into an enterprise sales leader. The guest shares lessons learned from driving 50,000 miles across the country, emphasizing customer problem-solving over consulting slide decks. |
Statements from this episode (21)
Ambati: Standing out in AI is harder because access is commoditized
“Having said that, I think it's also very difficult to get things right because the same three things are accessible to everyone. So really to stand out and then build something amazing in this day and age is going to be you know, exciting nevertheless, right?”
Ambati: Monthly AI model churn is fueled by synthetic data loops
“Every month, if you will, there is a new model, right? That's beating the old model, and visibly adopted already, and people are moving to the new models and the data is just flowing, like you're generating data, you're sort of getting access to new data creat…”
Ambati: Enterprise AI deals are won on trust, not model accuracy
“I would say enterprises don't optimize for accuracy of a model, right? So they're more so optimizing for reliability and trust. So it's very it's very, I think, in today's age with this new, ah, level of capability that the models have, getting the accuracy ri…”
Ambati: AI foundation models will capture more than 10% of software
“We are starting to see more evidence that models are going to own that more than the 10%, ah, you know, whether it's the new plugins that, ah, you know, Claude has released that shook up the whole SecOps market.”
Ambati: Trust-focused AI startups will beat incumbent software behemoths
“The advantage is towards startups who get this right. Because I can't imagine an existing behemoth going back and saying, okay, now I already have your trust in you know, reliability and everything. I can take care of that, but I'm going to reinvent and now be…”
Ambati: Advanced AI models will become more expensive as compute demands increase
“I think the, they're gonna get more expensive as the models get better the compute needs are gonna, you know, increase, and then you're gonna get better.”
Ambati: Base token prices will fall while latest frontier models get pricier
“So there is a time in the next year and two where we'll start to see these directionally, yes all the token prices are going down. But as the models are maturing, we are starting to see that, you know, the opposite effect as well, where, you know, the cost is …”
Ambati: AI inference market will reach $1.3T-$1.4T by 2030
“The inference market is one of the largest or so the fastest growing CAGR. So what we are seeing is about 94% CAGR. And then, you know, today, I think around 20, 26, we are looking at an eighty billion dollar sort of market. But soon by 2030 it's gonna be a 1.…”
Ambati: AI infrastructure spending is now driven primarily by inference
“The whole big bet of open AI, you know, trying to build these target and other You know, big data centers is not that they just want to train on this, because that cost is a one-time cost, right? So you train ones, you sort of run inference for a lifetime. So …”
Ambati: Current spike in AI voice and support automation will fade
“Anything that has a human-to-human touchpoint should be the last one that should get automated. Today we're just probably seeing that first, ah, sort of spike, but then that should probably, you know, die out soon, I feel.”
Ahluwalia: Sierra and Decagon are valued at nearly $10B
“Customer support has companies like Sierra, Decagan, Which are already almost ten billion dollar companies.”
Ambati: Sierra and peers will soon solve AI empathy and intervention
“And I'm betting that I'm, companies like Sierra and others will figure this out soon. But if they don't, then I think the spike has to come down.”
Ambati: Voice will be the frontier interface for interacting with AI
“Voice to me is still going to be the frontier through which people will interact with AI.”
Ambati: Enterprise healthcare software sales involve distinct buyer, payer, and user roles
“When you build something, you typically build with the CTO in mind, thinking I'm going to build this and I'm going to go and sell to the CTO. The CTO is where you sell, but then you actually get a check from CFO, where it's used by the CAO. So all of that mapp…”
Ambati: First enterprise customers should be chosen for credibility over revenue
“You need to think of your first customer, not in terms of revenue, but in terms of credibility, right? So they've added so much credibility to my tag that I could actually I didn't even have a startup at that point, but whatever I started afterwards, with this…”
Ambati: Walmart was Predera's first MLOps customer
“And, you know, funny enough that the first customer for our MLOps product was Walmart, the largest giant in, in retail. And, you know, they bought it not for themselves, but they wanted to open it up for all their suppliers.”
Ambati: Predera dropped Walmart contract in 2023 to pivot to LLMOps
“So, the boldest thing that we did was we did not renew the contract with Walmart, because they said, you know, continue the MLOps product, and we said, we'll be an LLM Ops company, and Walmart was not ready for LLM Ops at that point, but we said, okay, fine, w…”
Ambati: The core challenge in AI is workflow integration, not intelligence
“Today's AI you know, the problem of AI is not intelligence. But it's an integration problem, right? So you have to figure out how to integrate this into the right workflows, gain the right end users, and then sort of work with.”
Ambati: Palantir is a product company disguised as services
“It's a great product disguised as services. So it's, I'm sure there's merit to the product but the launching vehicle is services, right? Because these are hard to understand products that you have to take to the customer.”
Ambati: Do not pretend to build a product before five customers
“Until then, I wouldn't say, don't even pretend that you're building a product because that'll hurt you more than it'll help you. Because you're basically thinking that you're building product, the cycles go there, but then they'll really just start to look lik…”
Ambati: Enterprise middle managers distrust big consulting firm pitches
“Enterprises are actually looking for solutions to their problems. And quite often what happens is the big companies that they're working with, the managers, the VP and below levels, like the directors and managers, they actually don't approve of the, Call it t…”