The Exchanges

Q And which was your first job after UCLA?

A So I joined a very early post-acquisition Siri team at Apple, and the team was about 30, 35 people, still the founders were there, and, um, I ended up, it was a long story to, to, to join that team, but I ended up working there for Uh, a good number of years, about five years where I worked on the early knowledge graphs that Siri was built on. And, uh, it was kind of in the era of early deep learning models were being experimented. It was still very expensive to run it in production. Uh, but I kind of saw the whole arc of how do you build knowledge graphs and language learning systems without these models and then saw models kind of become a thing.

AI assessment note: “I joined a very early post-acquisition Siri team at Apple”

Q And, uh, you know, uh, Luna one was your first model, uh, that you built. Why did you build a model in the first place?

A Yes. So our Luna story is fascinating. So for, for those who don't know Luna, Luna is a small language model, which is specifically designed to solve the evaluation problem in AI. So how do you know whether, you know, an output is good or bad, or an input has some PII or any security issues. So these are very specific, uh, uh, tasks which Luna is designed to solve for. So by default, the size of the Luna model is much smaller. It is orders of magnitude smaller than, say, a general LLM, which is out there. In fact, our latest Luna models, which are some of our largest, they are, they typically range from one to three billion parameters, which is minuscule compared to some of the larger foundational models out there. Luna came into the picture when we realized that LLMs as judges, which was the traditional way that people were using to evaluate, and that has a whole history of it. Um, they don't scale. They don't scale in production. They don't allow you to do full scale observability, which means intercepting every single input and output of, you know, these AI systems. They simply choke. They are very expensive. And they're highly unoptimized to solve for the latency and cost problem. So we took this problem and kind of came back to the drawing board saying that, hey, how do we distill all this intelligence and reasoning abilities of LLMs and bring it down to a much smaller par…

AI assessment note: “Luna came into the picture when we realized that LLMs as judges... don't scale”

Q And how did you and Vikram zero in on the idea of Galileo?

A That was a long story. So we were discussing on what to build. Vikram was certainly very gung-ho about starting a company, and I was deeply in the AI space, so I knew that AI is the future, and I was very excited to build something in that. He also saw the same sort of vision. We initially started off working on some data infrastructure for machine learning systems, and While we were ideating and building, we realized that trust is a big problem in machine learning, and it is unsolved. So we started building a trust layer of every machine learning workflow. How do you quantify whether an output of an ML model is good or bad? And one thing led to another, and I created a prototype. We met a professor at Stanford, Chris Ray, who we knew through one of Vikram's friends, uh, who had recently sold a company that Chris Ray was involved in. So I showed him this little demo of a Jupyter notebook where you, the user publishes a model and before the model gets productionized, it would do these bunch of checks. And Chris Ray saw it and he's like, this is a great idea. We will do it for language models. And my only question to Chris was what the hell is a language model, right? Because LLMs were not, uh, you know, a word back then. And of course I knew NLP systems and stuff, but, uh, um, this idea of running language models at scale was so alien and so new. This is early, so late, but it s…

AI assessment note: “While we were ideating and building, we realized that trust is a big problem”

Q And, and assume latency, they'll also solve, uh, with time, right? Uh, now, even if you throw complex queries to clot today, uh, the results are usually under a minute.

A No, that's, uh, that's totally fair. So all these factors are going down, but despite that, there's some nuances in the, in, in the economics of this, which you'll see that it's, despite these improvements, the, the challenges around evals will still remain. The first reason for that is the more the usage of LLMs in the enterprise and the more gen AI adoption, uh, the infrastructural challenges will keep growing because even if the per unit latency reduces, more traffic means you're just, uh, subjecting the system to more, more tokens and just more data. And all that really brings the system down. That's one. Uh, number two is agent interactions are getting more and more complex. So even at a unit level, if, if say you apply some magical engineering and solve the problem of, hey, I reduced a 10:02, 15:02 complex generation to one second. Agentic interactions will compound.

AI assessment note: “even if the per unit latency reduces, more traffic means you're just”

Q So when software has become tokens, what does, what is the final definition of a great product be?

A That's a great question. I was figuring that out on my way here, and the day before that, and the week before that, that's pretty much all I think about. I can certainly share my, I guess, my learnings till now. Uh, it is, um, there's no, uh, no one knows the right answer per se. Uh, it is, what I've realized is that you have to meet people and workflows where they are, because it's not a clean sweep, clean canvas, right? There's, Software is pretty hairy. If you really look at what goes on behind the scenes, there's a lot of traditional software that's already in place, which solves really hard problems. So we have to fit into that and then evolve the form factor. Uh, the easiest thing to disrupt is what a good UI looks like in the new era. And I can certainly tell you about, um, in the observability and eval space, What is the problem that any observability tool really solves, whether it's AI or not AI? It's the, hey, what's wrong question. You ask a system, hey, what's wrong? And then you take 50 steps to figure out what's wrong. Traditionally, we have solved that through statistics, through charts, through drift, through all kinds of mathematical paradigms that we put into a software and that shows up like a chart, and that's what developers do. But now you're in the era where you can ask a question and get an answer, and there's all these steps in the middle which requires…

AI assessment note: “you have to meet people and workflows where they are... evolve the form factor”

Q Who are your first customers, and what's the first use case that you solved for, let's say the first 10 customers?

A Yeah, so given we started before LLMs were a thing, a lot of the workflows, especially in the enterprise, was driven around fine-tuning, Uh, some of the smaller language models. So for those who've only learned of language models after ChatGPT, they wouldn't know about the erstwhile models, but these models were, were already there. In fact, even before GPT 3.5, you had GPT two and GPT one, which were essentially similar architectures as ChatGPT, the, the modern ChatGPT, except the number of, um, the size was much lower. And smaller. And in, in the AI world, we measure the size of the models through what we call the number of parameters. So if we ever hear people say, uh, I have a five hundred billion parameter model, they are basically saying they have a really large model. So coming back to the workflows, folks were working with much smaller models back then, typically in the hundred to four hundred million parameter range. So very small size, It's so small that in theory you could run them on CPUs. They would be slow because it's sequential, but the GPU infrastructure also needed to serve these models was much less resource intensive. And the workflows back then were primarily around contact center AI and entity detection. And these were, this is what we call tasks. So entity linking, entity detection, classification, These were standard language modeling tasks that enterpri…

AI assessment note: “workflows back then were primarily around contact center AI and entity detection”