Feb 5, 2025 · 31m · latent-space
Why every AI Engineer needs an AI Gateway (ft Portkey.ai CEO)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Portkey.ai CEO Rohit Agarwal joins hosts Swix and Alessio to discuss the critical role of AI gateways in managing model routing, production guardrails, OpenTelemetry observability, and multi-step agentic workflows.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 5.9 Guest teaching 3.0 Guest disagreement 1.2 The hosts pushing back 3.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 14:08 Contesting automated gateway retries
Swix directly challenges Portkey's custom status code retry mechanism, arguing that dumb retries fail without the application's unique prompt context and tools.
Hardest push from the hosts ▶ 28:06 Challenging the novelty of MCPSwix pushes back on the hype around MCP, questioning why it is revolutionary rather than just a minor variation of the existing OpenAPI specification.
Biggest teaching moment ▶ 29:07 Two-way communication in MCP specRohit explains the bidirectional RPC and SSE capabilities of MCP, prompting Swix to openly admit he had missed that entire capability despite browsing the specification.
The host holds their own ▶ 2:12 Reasoning model latency driving routing needsAlessio demonstrates industry authority by bringing fresh findings from a Fortune 500 enterprise council to illustrate exactly where routing is becoming critical due to reasoning model latency.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Model Routing for Reasoning LLMs and Agentic Systems | 7 | 1 | 1 | 4 | Alessio demonstrates strong domain knowledge by challenging the necessity of routing in single-model setups and sharing concrete insights from a Fortune 500 AI council regarding reasoning model latencies. Rohit readily validates Alessio's framing and builds on it with recent DeepSeek data. | |
| Decoupling Gateway Architecture from Application SDKs | 6 | 4 | 1 | 3 | Swix expresses strong architectural opinions against putting LLM abstractions in application code while raising valid concerns about proxy latency and downtime. Rohit clarifies his architecture by explaining their 21MB JSON transformer engine, which Swix clarifies isn't an LLM transformer. | |
| Core Gateway Capabilities: Observability, Limits, and Governance | 4 | 3 | 0 | 1 | Rohit provides a structured overview of core gateway functions including rate limits, budget enforcement, guardrails, and enterprise chargebacks. The hosts act primarily as facilitators guiding the categorization. | |
| Production Guardrail Implementation and Error Orchestration | 7 | 2 | 2 | 6 | Alessio opens with historical context on guardrail evolution, and Rohit explains practical regex versus demo PII guardrails and custom status code 246. Swix directly pushes back on gateway-level dumb retries, arguing that retry prompts require application-level context and tool awareness. | |
| AI Observability and the Evolution of OpenTelemetry Standards | 6 | 2 | 1 | 4 | Swix probes Portkey's positioning against traditional observability vendors and standard bodies like OpenLLMetry. Both Swix and Rohit note that standardization efforts often lag behind rapidly shifting API paradigms. | |
| Observability and Debugging in Multi-Step Agent Architectures | 6 | 2 | 2 | 5 | Rohit details the debugging requirements and routing opportunities in multi-step agents. Swix counters with skepticism regarding multi-LLM setups, arguing that switching models complicates prompts and accumulates endpoint downtime risks. | |
| Real-Time Production Evals and Aligning Human Feedback | 5 | 3 | 1 | 2 | Swix candidly admits that developers and end users rarely provide manual human feedback. Rohit agrees and shares successful alternative patterns where implicit business metrics (like video download rates) serve as positive feedback signals. | |
| Defining Traces and Sessions in Ambient and Continuous AI | 7 | 3 | 1 | 3 | Rohit explains the OpenTelemetry model of traces versus sessions, but Swix challenges the paradigm by introducing continuous wearable audio streams and ambient agents where clear session boundaries do not exist. | |
| Model Context Protocol (MCP) and Gateway Tool Integration | 5 | 7 | 2 | 5 | Swix questions why MCP is seen as revolutionary compared to OpenAPI specs. Rohit educates Swix on two-way communication capabilities and RPC mechanisms, leading Swix to acknowledge he did not realize two-way agent functionality existed in the spec. |
Statements from this episode (15)
Agarwal: AI gateways act as operational platforms offering governance beyond basic proxies
“So I think AI gateways are essentially the operational platforms that enable teams to connect to LLMs more efficiently. They help you improve cost, performance, and accuracy by not having you to build individual connections to all of these different AI service…”
Agarwal: 90% of production LLM use cases do not use automatic routing
“In fact, I would say in production, 90% of the use cases do not use automatic routing. What they want is deterministic flows. As long as the gateway manages authentication authorization for them, it's perfectly fine. The request hitting A specific model that t…”
Agarwal: Multi-step agentic systems are driving adoption of LLM routing
“With the rise of agentic systems where people are building multi-step agents, that's also where routing is becoming really popular because you might want to have different tasks point to different LLMs, and then you don't want to build authorization and all of…”
Agarwal: Portkey open source gateway has a 21 MB memory footprint
“So the gateway, when it runs, has a memory footprint of like, 21 MB, and the way we've been able to do it is just using a very interesting architecture where we're using transformers in JSON to do everything out of the box.”
Agarwal: Common AI guardrails focus on format and topics over PII
“When I talk about guardrails, the first thing that comes to mind is, oh yeah, we have to do PII redaction and sensitive data redaction. But somehow we've seen that the more common use cases for guardrails end up being, you know, is it the right length? Is it u…”
Agarwal: Gateway-based guardrails reduce latency versus application-level checks
“And again, you can decrease latency massively by building this on the gateway. Rather than building this within your application.”
Agarwal: Regex rules are the most deployed LLM guardrails in production
“In reality, we've seen what gets deployed the most is regex-based guardrails to say, you know, I want to catch for specific words, and then take certain actions based on that. Or I want to catch for empty outputs.”
Agarwal: Gateways produce the best LLM metrics without manual codebase instrumentation
“We're saying that the best LLM metrics can be produced directly on the gateway. Without every team having to instrument their own code bases to pick out these metrics.”
Agarwal: AI telemetry cannot standardize while underlying APIs change monthly
“Standards usually evolve when there's some amount of coherence and stability in an API. I think the APIs themselves change every month. So I'm not sure how there's going to be standardization that occurs.”
Agarwal: AI teams rarely use human feedback, despite high satisfaction when used
“I think it's not used as much. I agree. And I think that's something I tell every customer that you need to close the loop and feedback is going to help you. I think the teams that are doing it are really happy. They're not doing it.”
Agarwal: LLM teams will adopt human feedback only after stabilizing applications
“I think the output side of things will come. Once we've reached stable state, where there are production applications that are stable, and then they want to start optimizing, is when they'll start looking at human feedback as well.”
Agarwal: Successful LLM feedback loops map directly to business metrics
“I think the teams that have been successful is when they're mapping these feedback metrics to a business metric that they track.”
Agarwal: Ambient AI agents will break traditional fixed-boundary telemetry traces
“I would imagine like with more ambient agents coming in, there might not be a clear start and a clear end. So, it'll be interesting to see how we start solving this problem. Maybe the whole concept of, you know, there's a typical start and a typical end goes a…”
Agarwal: MCP will become standard way agents connect to services
“I think the future of agents, the way they connect to different services is going to be MCP.”
Agarwal: AI production will require operational tools just like DevOps enabled cloud
“The more they want platforms like this, and I feel this is similar to what happened with the cloud era back in 2012. You will not have cloud adoption till the time DevOps did not take off, and companies like Datadog, Cloudflare, et cetera, didn't help you buil…”