Assertion Not checkable as stated
Agarwal: 90% of production LLM use cases do not use automatic routing
“In fact, I would say in production, 90% of the use cases do not use automatic routing. What they want is deterministic flows. As long as the gateway manages authentication authorization for them, it's perfectly fine. The request hitting A specific model that t…”
Assertion Not checkable as stated
Agarwal: Gateway-based guardrails reduce latency versus application-level checks
“And again, you can decrease latency massively by building this on the gateway. Rather than building this within your application.”
Insight
Agarwal: Gateways produce the best LLM metrics without manual codebase instrumentation
“We're saying that the best LLM metrics can be produced directly on the gateway. Without every team having to instrument their own code bases to pick out these metrics.”
Insight
Agarwal: AI telemetry cannot standardize while underlying APIs change monthly
“Standards usually evolve when there's some amount of coherence and stability in an API. I think the APIs themselves change every month. So I'm not sure how there's going to be standardization that occurs.”
Insight
Agarwal: Successful LLM feedback loops map directly to business metrics
“I think the teams that have been successful is when they're mapping these feedback metrics to a business metric that they track.”
Prediction Not checkable as stated
Agarwal: MCP will become standard way agents connect to services
“I think the future of agents, the way they connect to different services is going to be MCP.”
Insight
Agarwal: AI gateways act as operational platforms offering governance beyond basic proxies
“So I think AI gateways are essentially the operational platforms that enable teams to connect to LLMs more efficiently. They help you improve cost, performance, and accuracy by not having you to build individual connections to all of these different AI service…”
Insight
Agarwal: Common AI guardrails focus on format and topics over PII
“When I talk about guardrails, the first thing that comes to mind is, oh yeah, we have to do PII redaction and sensitive data redaction. But somehow we've seen that the more common use cases for guardrails end up being, you know, is it the right length? Is it u…”
Assertion Not checkable as stated
Agarwal: Regex rules are the most deployed LLM guardrails in production
“In reality, we've seen what gets deployed the most is regex-based guardrails to say, you know, I want to catch for specific words, and then take certain actions based on that. Or I want to catch for empty outputs.”
Assertion Not checkable as stated
Agarwal: AI teams rarely use human feedback, despite high satisfaction when used
“I think it's not used as much. I agree. And I think that's something I tell every customer that you need to close the loop and feedback is going to help you. I think the teams that are doing it are really happy. They're not doing it.”
Prediction Not checkable as stated
Agarwal: LLM teams will adopt human feedback only after stabilizing applications
“I think the output side of things will come. Once we've reached stable state, where there are production applications that are stable, and then they want to start optimizing, is when they'll start looking at human feedback as well.”
Prediction Not checkable as stated
Agarwal: Ambient AI agents will break traditional fixed-boundary telemetry traces
“I would imagine like with more ambient agents coming in, there might not be a clear start and a clear end. So, it'll be interesting to see how we start solving this problem. Maybe the whole concept of, you know, there's a typical start and a typical end goes a…”
Prediction Not checkable as stated
Agarwal: AI production will require operational tools just like DevOps enabled cloud
“The more they want platforms like this, and I feel this is similar to what happened with the cloud era back in 2012. You will not have cloud adoption till the time DevOps did not take off, and companies like Datadog, Cloudflare, et cetera, didn't help you buil…”
Insight
Agarwal: Multi-step agentic systems are driving adoption of LLM routing
“With the rise of agentic systems where people are building multi-step agents, that's also where routing is becoming really popular because you might want to have different tasks point to different LLMs, and then you don't want to build authorization and all of…”
Assertion Partly supported
Agarwal: Portkey open source gateway has a 21 MB memory footprint
“So the gateway, when it runs, has a memory footprint of like, 21 MB, and the way we've been able to do it is just using a very interesting architecture where we're using transformers in JSON to do everything out of the box.”
