Soham Ganatra, co-founder of Composio, shares metrics on the size of Composio's application catalog.
Opinion
Ganatra: Existing AI tool-calling benchmarks are not production-ready
“Tool calling has a lot of different evals. You have seen the function tool calling evals. You have multi-function, like, multi-tune tool calling evals. But, like, none of them are actual, like, production quality. Like, none of them are actually something that…”
Assertion Not checkable as stated
Ganatra: 95% of Composio integrations are built and maintained by agents
“So, 95% of all our integrations are completely built using agents, maintained using agents.”
Assertion Supported
Ganatra: MCP Lacks Client Post-Processing for Long Context Responses
“MCP let's just say you have a tool that has really long context, a really long sort of response. Currently, if you use MCP for those kinds of tools, it ends up sort of messing up, right? Because let's say I'm trying to fetch like a hundred emails on Gmail and …”
Assertion Supported
Ganatra: Server-Controlled Schemas in MCP Complicate Client-Side Customization
“Some of the other things that we have observed is like, hey, maybe I want to sort of change the tool descriptions, but like in MCP, the tool descriptions are controlled by the server itself. And so essentially for me as a developer who's sort of using an MCP, …”
Assertion Not checkable as stated
Ganatra: Over 100,000 developers have built with Composio
“In terms of developers, we have, like, more than 100,000 developers who have been building on top of course and, like, trying our product out.”
Assertion Not checkable as stated
Ganatra: Poor API documentation is the main source of tool-calling errors
“Is this Salesforce tool getting transformed into the right request payload so that it is hitting the right endpoint in Salesforce, right? And that is like the major source of error today, because a lot of documentations are not great. A lot of like ad cases ar…”