Agarwal: LLM teams will adopt human feedback only after stabilizing applications
Rohit Agarwal · Why every AI Engineer needs an AI Gateway (ft Portkey.ai CEO) · Feb 5, 2025 · at 22:26
Rohit Agarwal is the CEO of Portkey.ai, explaining the adoption curve of LLM observability and evaluation tools.
0:00 / 0:11exact quote · 11.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think the output side of things will come. Once we've reached stable state, where there are production applications that are stable, and then they want to start optimizing, is when they'll start looking at human feedback as well.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Rohit Agarwal
Agarwal: 90% of production LLM use cases do not use automatic routing
“In fact, I would say in production, 90% of the use cases do not use automatic routing. What they want is deterministic flows. As long as the gateway manages authentication authorization for them, it's perfectly fine. The request hitting A specific model that t…”
Agarwal: Gateway-based guardrails reduce latency versus application-level checks
“And again, you can decrease latency massively by building this on the gateway. Rather than building this within your application.”
Agarwal: Gateways produce the best LLM metrics without manual codebase instrumentation
“We're saying that the best LLM metrics can be produced directly on the gateway. Without every team having to instrument their own code bases to pick out these metrics.”
Agarwal: AI telemetry cannot standardize while underlying APIs change monthly
“Standards usually evolve when there's some amount of coherence and stability in an API. I think the APIs themselves change every month. So I'm not sure how there's going to be standardization that occurs.”