Agarwal: AI production will require operational tools just like DevOps enabled cloud
Rohit Agarwal · Why every AI Engineer needs an AI Gateway (ft Portkey.ai CEO) · Feb 5, 2025 · at 30:03
Rohit Agarwal is the CEO of Portkey.ai, discussing the evolution of enterprise AI infrastructure with hosts Swix and Alessio.
0:00 / 0:44exact quote · 44.5s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The more they want platforms like this, and I feel this is similar to what happened with the cloud era back in 2012. You will not have cloud adoption till the time DevOps did not take off, and companies like Datadog, Cloudflare, et cetera, didn't help you build applications, you know, for faster, cheaper, more efficient. I think the same is going to happen in AI as well. As more and more teams go to production, you need tools like You know, not just the AI gateway, but multiple other operational efficiency tools that will help you stay reliable, efficient in production. I think that's something that I'm looking forward to in 25, 26 because we're seeing a lot more teams finishing their POC cycles and moving into production with AI.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Rohit Agarwal
Agarwal: 90% of production LLM use cases do not use automatic routing
“In fact, I would say in production, 90% of the use cases do not use automatic routing. What they want is deterministic flows. As long as the gateway manages authentication authorization for them, it's perfectly fine. The request hitting A specific model that t…”
Agarwal: Gateway-based guardrails reduce latency versus application-level checks
“And again, you can decrease latency massively by building this on the gateway. Rather than building this within your application.”
Agarwal: Gateways produce the best LLM metrics without manual codebase instrumentation
“We're saying that the best LLM metrics can be produced directly on the gateway. Without every team having to instrument their own code bases to pick out these metrics.”
Agarwal: AI telemetry cannot standardize while underlying APIs change monthly
“Standards usually evolve when there's some amount of coherence and stability in an API. I think the APIs themselves change every month. So I'm not sure how there's going to be standardization that occurs.”