Real-time AI apps require abstraction layers to handle LLM provider outages
Ali Dasdan · AI at ZoomInfo: Superpowering GTM teams | Ali Dasdan, CTO, ZoomInfo · Sep 19, 2024 · at 38:00
Ali Dasdan is CTO at ZoomInfo. He explains the operational necessity for ZoomInfo to build an internal middleware layer above LLM providers.
0:00 / 0:25exact quote · 25.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Yeah, I think you have to create a layer on top of these at a minimum, right in the end, whether Entropic or OpenAI, you know, they're companies, right, they sometimes have stability issues, right, if they're down, so what they're gonna do about that? Since our responses co-pilot is all real-time, we have to respond to that. There is security. That whatever we are sending and whatever we are getting as a result of that has to be scanned, right? So that layer actually includes that.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ali Dasdan
Dasdan's past teams built custom Kafka alternatives to handle 100 petabytes
“In the past, when we were doing data platform, there was no Kafka, for example, we created a Kafka-like system ourselves, but we are lucky that, yeah, we created an HBase, sort of HHive-like system ourselves with hundred petabytes of size of data, right?”
ZoomInfo claims its homegrown NLP outperforms major external vendors on quality
“Chorus still runs its own, our own homegrown NLP technology. We are doing it because our quality, we believe every time we compare, it's actually pretty good. Actually, we are beating, you know, we've been well-known well-known vendors. Plus it's cheap for us,…”
Adding generative AI call summarization boosted ZoomInfo Chorus's NPS by 20 points
“And the absolute increase in MPS was just amazing, like almost like 20 points. Because people loved what they were getting with that.”
ZoomInfo engineers accept over 25% of GitHub Copilot's code suggestions
“And our acceptance rate for code suggestions was around 25%, actually over 25%, which is interesting in line with the recent research that GitHub published.”
ZoomInfo operates two distinct core data platforms on different technologies
“On top of it we have actually two data platforms. One is for all the data that customers, you know, need and our products need, and there's another one that's for internally focused for our internal customers, and they're built on different technologies.”