Mar 19, 2026 · 1h 18m · neon-show
Which AI Model are companies actually Paying For in 2026? | Rohit Agarwal, Portkey
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this in-depth discussion on the Neon Show, Portkey founder Rohit Agarwal and host Siddharth Ahluwalia examine the scaling of enterprise AI infrastructure, detailing how centralized AI gateways, agentic workflows, and premium foundation models are reshaping software economics. Agarwal breaks down the practical mechanics of production LLM deployments, from managing multi-million-dollar API spending to building defensible vertical AI architectures.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
Siddhartha as informed peer 2.8 Guest teaching 3.9 Guest disagreement 0.3 Siddhartha pushing back 0.2
speaking balance: gold is Siddhartha, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 30:00 Rohit rejects cost-obsession framing
Rohit dismisses the common market narrative that companies solely want cheap LLMs, presenting hard telemetry that the costliest models are seeing the highest token usage.
Hardest push from Siddhartha ▶ 35:35 Host pushes back on irrational SaaS stock dumpingThe host forcefully challenges market panic, arguing that frontier lab release hype unfairly tanks public enterprise SaaS stocks despite practical deployment realities.
Biggest teaching moment ▶ 1:09:45 Rohit explains enterprise security threats of MCPRohit educates the host on how unauthorized local MCP servers in developer environments create critical corporate security vulnerabilities without a central gateway.
Siddhartha holds their own ▶ 49:20 Host articulates vertical domain moat thesisThe host presents an elaborate investor thesis explaining why deep institutional knowledge and vertical client networks protect startups from general foundation labs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Siddhartha as informed peer | Guest teaching | Guest disagreement | Siddhartha pushing back | Why |
|---|---|---|---|---|---|---|
| Episode Preview and Highlights on the Future of Enterprise AI | 1 | 1 | 0 | 0 | Introductory trailer and welcoming exchange between the host and guest, setting up the episode premise without technical debate. | |
| Demystifying the Three-Layer Enterprise AI Architecture Stack | 2 | 5 | 1 | 0 | The host asks Rohit to explain the AI stack for beginners; Rohit delivers an educational breakdown comparing compute, operational, and application harness layers. | |
| Origins of Portkey and Direct Parallels to the DevOps Movement | 2 | 4 | 0 | 0 | Rohit shares war stories from previous startups that inspired Portkey, drawing a direct parallel between DevOps adoption and AI operations. | |
| The AI Gateway Landscape and Emerging Enterprise Reference Architecture | 2 | 4 | 0 | 0 | Discussion of competing AI gateways and Portkey's strategic decision to focus on centralized production services rather than developer SDKs. | |
| Simplifying Multimodal Complexity Across Unified LLM Endpoints | 2 | 4 | 0 | 0 | Rohit explains technical complexities in handling streaming, multimodal inputs, and unified endpoints across diverse model providers. | |
| Deconstructing Agent Harnesses, Autonomous Loops, and Claude Market Hype | 2 | 5 | 1 | 0 | The host asks Rohit to define agent harnesses and clarify market hype; Rohit explains the mechanics of continuous tool-calling execution loops. | |
| Deploying Autonomous AI Employees: SagePilot Case Study and Workflow Automation | 4 | 3 | 0 | 0 | The host details a portfolio case study of SagePilot replacing support staff; Rohit analyzes context window retention and agent management dynamics. | |
| Scaling Global AI Traffic and Prioritizing Token Volume over Lagging Revenue | 2 | 4 | 1 | 0 | Rohit details scaling token volume over immediate monetisation, dismissing macro valuation concerns while reaffirming core underlying utility. | |
| Founder Productivity Playbook: Practical Workflows for Internal Automation | 2 | 3 | 0 | 0 | Rohit gives concrete examples of automating personal founder workflows across bug fixing, contract reviews, and compliance tasks using AI assistants. | |
| The Shift to High-Value Models: Enterprise Prioritization of ROI Over Token Cost | 3 | 5 | 1 | 0 | Rohit explains how enterprise usage patterns shifted from cheap mini models toward expensive frontier reasoning models like Claude Opus. | |
| Enterprise AI Adoption Curves: From Software Engineering to Core Business Functions | 2 | 4 | 0 | 0 | Rohit outlines how enterprise AI adoption expands chronologically from software engineering into knowledge work, HR, and legal operations. | |
| Macroeconomic Impact of Frontier Lab Releases on Legacy Public SaaS Stocks | 6 | 3 | 1 | 2 | The host posits that frontier lab opacity and release hype create irrational macroeconomic sell-offs in legacy public SaaS stocks; Rohit contextualizes this as a forcing function for enterprise innovation. | |
| The Rise of Technical Founders, Modern Distribution, and AI-Augmented Sales | 4 | 3 | 0 | 0 | Discussion on how technical founders can scale enterprise sales motions with lean teams before bringing in dedicated GTM leadership. | |
| Anthropic's Cognitive Lead, Natural Tone, and the Disruption of Vibe Coding | 3 | 4 | 1 | 1 | Discussion on Claude's natural conversational tone and Anthropic directly threatening standalone vibe coding and legal tech applications. | |
| Global AI Consumption Trends, Regional Hotspots, and Emerging Voice Modalities | 2 | 4 | 0 | 0 | Rohit maps international traffic distribution, highlighting US dominance, unique Australian banking innovation, and rapid voice growth in India. | |
| Defensibility Through Institutional Domain Knowledge and Founder Agility | 7 | 2 | 0 | 2 | The host articulates an in-depth thesis on vertical defensibility via deep institutional domain knowledge; Rohit agrees and highlights early-stage founder agility. | |
| The Strategic Imperative of AI Neutrality and Gateway Independence | 3 | 5 | 0 | 0 | Rohit explains how neutral gateway infrastructure prevents vendor lock-in, enabling enterprise clients to switch providers seamlessly via a single toggle. | |
| Next-Gen AI Gateway Roadmap: Enterprise Daemon and Usage Intelligence | 2 | 4 | 0 | 0 | Rohit details Portkey's roadmap, including the local enterprise gateway daemon and mining usage intelligence across production API calls. | |
| The Transformation of Legacy SaaS: From Interaction Portals to Systems of Record | 3 | 5 | 1 | 0 | Rohit argues traditional SaaS platforms will be reduced to background systems of record while autonomous agentic harnesses become the primary execution layer. | |
| Navigating the Build versus Buy Dilemma and Enterprise Budget Constraints | 3 | 4 | 0 | 0 | Discussion on how code generation tools and constrained IT budgets are temporarily tilting the enterprise build-versus-buy calculation towards internal builds. | |
| Managing $180M in Production LLM Spend and the Unit Economics of AI APIs | 3 | 5 | 0 | 0 | Rohit contrasts traditional zero-cost API requests with high-value LLM API calls, illustrating how Portkey manages $180M in customer token spend. | |
| The Model Context Protocol Revolution, Tool Calling, and Agentic Orchestration | 3 | 5 | 0 | 0 | Rohit breaks down Model Context Protocol (MCP), tool calling orchestration, and why enterprise security demands centralized governance over external MCP servers. | |
| Open Core Strategy, Developer Trust, and Lessons from Parag Agrawal | 2 | 5 | 0 | 0 | Rohit recounts advice from Parag Agrawal that led Portkey to adopt an open-core architecture to build authentic developer trust. | |
| Open Source Milestones, Industry Validation, and Podcast Conclusion | 1 | 2 | 0 | 0 | Rohit shares open-source traction metrics and community feedback, followed by mutual closing remarks and podcast wrap-up. |
Statements from this episode (37)
Agarwal: Claude Code and Cursor act as harnesses for underlying models
“There's a new class of these applications called harnesses, which are becoming very popular, so look at ClaudeCode, Cursor. They're essentially harnesses in a way that they're applications, but they help you use the underlying models a lot better.”
Agarwal: Portkey has nearly 500,000 open-source installs
“So we've got a very large open source footprint. That's almost a half a half a million installs of the open source system across companies.”
Agarwal: Visibility and spending control are enterprises' two biggest AI anxieties
“The two emotions people are very worried about in the AI world, especially larger enterprises, is I don't have visibility on what's happening. It's a black box. It goes wrong. It hallucinates. It can cause problems. My agents can break a lot of these things, r…”
Agarwal: A bot attack drained Pepper's monthly AWS budget in one day
“We had an attack one day where we ended up spending our month's AWS bill in just one day on OpenAI because we had a bot attack happen on us.”
Rohit Agarwal: Retool is no longer very popular
“Retool is no longer very popular, but we were heavy users of it, like in 20, 20, 21.”
Agarwal: Production enterprise AI standardizes on three core architectural layers
“If I'm building my AI systems, I need my LLMs, I need my databases I need my operational platform, which is the control plane, like Portkey, and then I need my application harnesses on top. So this is almost like a very simplified enterprise AI architecture, b…”
Agarwal: Production software infrastructure makes more money than developer tools
“Production companies make money and are And see just much higher adoption than developer-focused companies.”
Agarwal: AI operational spend will surpass traditional cloud DevOps spend
“So will AI spend surpass cloud spend? I a hundred percent believe that, which means that the AI ops spend surpasses DevOps spend. It's bound to happen.”
Agarwal: Continuous tool loops deliver exponential value over LLM Q&A
“So that's the simplest harness, which is make an LLM call, make a tool call, and then make the loop again, which is just a continuous loop. And that alone is so much more powerful than just a simple Q and A because it's now taking actions. It's telling you nex…”
Agarwal: 80% of engineering tasks can be automated with AI
“80% of this work, I think can be automated with AI.
I still believe you need the last 20%.
So there still has to be a manager of these agents who's telling you,
Okay.
Do this.
Don't do this.
but outside of that, 80% of the work is still getting done fa…”
Ahluwalia: Apparel firm cut support staff from 16 to 3 using SagePilot
“One of the customers of Sage Pilot, like the 50 crore revenue company based in Jaipur they do clothes clothing, right?
Women's clothing.
And they mentioned that
They have reduced their team of 16 people in customer support and customer lifecycle to three pe…”
Agarwal: Portkey handles roughly 0.5% of global AI API traffic
“I would assume we are probably .5% of total AI API traffic today.”
Agarwal: Portkey routes 15 trillion tokens daily, up from 1B
“So I think we were probably doing a billion tokens a day a year back. Now we're doing 15 trillion.”
Agarwal: Existing AI components suffice to build better systems without new innovation
“I feel that today, even if all innovation were to stop, there is enough fuel in the AI space to see you can still continue to Pick up these Lego pieces and still keep building newer and better systems.”
Agarwal: The AI bubble is in valuations, not utility
“So the bubble might be on the valuations. The bubble is not on utility.”
Agarwal Uses AI Assistant to Debug and Commit Code While on Calls
“Slowly from then on, it went into, hey, can I start solving software bugs through the same assistant as well? That started happening. So my team started seeing me making more commits and they're like, how are you on calls and still making commits here? Because…”
Agarwal: Claude Opus 4.6 leads Portkey volume despite being the costliest model
“For the first time, we saw four, six Opus, which is the costliest model there is right now, is at the top of the charts in terms of number of tokens spent.”
Agarwal: AI Buyers Prioritize ROI Over Token Cost, Embracing Premium Pricing
“People are no longer worried about cost of AI. They're worried about return of investment, which is like, okay, I'm happy to spend 203 hundred dollars a month on Claude code, but the value that I get from it is much, much higher.”
Agarwal: Engineering and consulting proposals drive top usage on Portkey
“And we see, obviously again, software development, engineering teams using it is the highest. Knowledge work is the second, which is large consulting teams or large business teams that are preparing proposals, et cetera.”
Agarwal: Claims that 80% of code is AI-written are accurate
“There are so many companies coming out and saying, 80% of all of my code base today is written by AI, and that is truth, and that's what we're seeing everywhere as well.”
Agarwal: 20% stock drops after AI announcements force legacy SaaS into agents
“ServiceNow and Salesforce and Workday and all of these companies are being pushed to innovate. Like in other cases, they're like, we have large go to market motions. There's a revenue that we can easily protect and you don't have to, but because they see these…”
Agarwal: Cheap Software Creation Demands Systems Thinkers Over Pure Executors
“Because building software has become so cheap, you need strong product thinkers or systems thinkers rather than just strong executors.”
Agarwal: Codex is better at writing code, but Claude leads in planning
“Everybody's still saying that Codex is a better model for code generation today, but people still use Claude to think, brainstorm, plan, and then once the plan is ready, it's like, okay, now Codex implemented. Codex is much better at writing code.”
Agarwal: Anthropic's Claude Code is killing the standalone vibe coding startup market
“It is killing that market. I think there is, that market is now finding its own niches, and Anthropic clearly realized that if 50% of my API revenue is going to come from vibe coding, they will Claude code and be like, I can still build the best vibe coding sy…”
Agarwal: Vertical AI survives because specialized niches aren't frontier labs' priority
“In their top three priorities, killing Harvey will not be their top three priorities. I mean, they're like, how do I make the next hundred billion in revenue? So, I mean, that is their thought process. So then Harvey can always be like, okay, then my top goal …”
Agarwal: AI startups must prove they offer better returns than backing Anthropic
“And I've heard this in conversations where a VC would be like, do you, do I invest in startups, which is like a risky business and I'm still getting a 10 X return or do I just put my money in Anthropic and I'll just get a 10 X return without doing any work. So…”
Agarwal: Voice AI has grown faster than text in India
“I feel in India, voice has taken off a lot faster than text. In other geographies, text and other modalities still dominate.”
Ahluwalia: Domain institutional knowledge will surpass frontier AI models
“In, in my view, where a startup can at least have a path to a hundred million in revenue, and this is still theoretical here to get played out in practice, just from observations from our portfolio company executing, is institutional knowledge in your domain w…”
Agarwal: Software is bifurcating into model labs and traffic control planes
“Almost the software world is getting divided into two. Either you can be an AI lab, which is the intelligence, you're building models, you're doing new stuff. Or you have control over this traffic so that you are doing the harness, application, ops, all of thi…”
Agarwal: Conversational record updates will replace manual data entry within two years
“This year, next year, you are going to be like, spoke to this person, this is what happened, update my recall. I think that is the world that is like very real. I think it will happen.”
Agarwal: SaaS will become systems of record while AI agents handle execution
“In the current format, SAS companies will just become systems of records. So you had the database, which got obsolete because of a SAS platform, which is just crud on this database, but interesting crud on this database. Now, SaaS platforms I think are the new…”
Agarwal: AI coding tools are pushing enterprises to build rather than buy
“Everybody is now saying, let's build first instead of buying first.”
Agarwal: Enterprise software budgets have stalled entirely outside of AI initiatives
“Like if you're going and asking for a 100,000 budget or a 500,000 budget, except AI, there is no spend happening in any other software category.”
Agarwal: AI gateways manage material per-request costs unlike traditional API gateways
“Every API request is literally that has a dollar value to it. Like an Opus request with 10,000 tokens is probably 23 cents, and this is, you're making a thousand requests, which means this is 23 dollars worth of API requests that you've made today, or two 30 d…”
Agarwal: Portkey saw customer LLM API spend grow 10x in four months
“And we've almost, I think, on Tenex in LLM spend in just four months. And that's because we've seen.”
Agarwal: Gemini 2.5 Pro and Claude Opus have overtaken OpenAI models
“Our top two models used to be GPT-Foro, GPT-Five Mini, plots on it. This used to be a top three for a long period of time. Now our top three averages checking between was, is Gemini 2.5 pro, Opus 4.6, Sonnet.”
Agarwal: Enterprises cannot safely deploy unmanaged MCP servers to production
“You can't have an MCP server in production because security team is coming and telling you, oh, what if this MCP server is malicious tomorrow? How do I shut it down? Because I now have thousand engineers all using this MCP server on their cloud code. I can't g…”