Agarwal: Gateways produce the best LLM metrics without manual codebase instrumentation
Rohit Agarwal · Why every AI Engineer needs an AI Gateway (ft Portkey.ai CEO) · Feb 5, 2025 · at 15:01
Rohit Agarwal, CEO of Portkey.ai, discusses Portkey's approach to AI observability and how gateway-level metrics contrast with manual instrumentation.
0:00 / 0:09exact quote · 9.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We're saying that the best LLM metrics can be produced directly on the gateway. Without every team having to instrument their own code bases to pick out these metrics.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Rohit Agarwal
Agarwal: 90% of production LLM use cases do not use automatic routing
“In fact, I would say in production, 90% of the use cases do not use automatic routing. What they want is deterministic flows. As long as the gateway manages authentication authorization for them, it's perfectly fine. The request hitting A specific model that t…”
Agarwal: Gateway-based guardrails reduce latency versus application-level checks
“And again, you can decrease latency massively by building this on the gateway. Rather than building this within your application.”
Agarwal: AI telemetry cannot standardize while underlying APIs change monthly
“Standards usually evolve when there's some amount of coherence and stability in an API. I think the APIs themselves change every month. So I'm not sure how there's going to be standardization that occurs.”
Agarwal: Successful LLM feedback loops map directly to business metrics
“I think the teams that have been successful is when they're mapping these feedback metrics to a business metric that they track.”
Agarwal: MCP will become standard way agents connect to services
“I think the future of agents, the way they connect to different services is going to be MCP.”
Agarwal: AI gateways act as operational platforms offering governance beyond basic proxies
“So I think AI gateways are essentially the operational platforms that enable teams to connect to LLMs more efficiently. They help you improve cost, performance, and accuracy by not having you to build individual connections to all of these different AI service…”