The Exchanges

Q Okay, super great. So to unpack, um, some of this, uh, so you mentioned retrieval augmented Fine tuning. Uh, is that, is that the same thing or is that different from, uh, retrieval augmented generation? Is that in terms or different?

A Uh, great question. They are two different things. So we, we have, you know, simple SDKs on top of our system for RAG, retrieval augmented generation, which is very common, um, these days. Um, retrieval augmented fine tuning is taking that and to the next, next level and to incorporate that into the training process. So retrieval augmented generation, RAG, which is very commonly used today, is a way to actually get information in, um, at inference time, um, during prompt engineering, essentially. Um, but for the model to learn new knowledge, retrieval augmented fine-tuning is actually incorporating that retrieval technology into the fine-tuning process. And something that I'm very, very excited about is These two very big communities kind of bring being moved together. So one community is, you know, the AI large language model community, and the other community has, you know, decades of research on information retrieval. Um, and I'm very excited to, you know, see these two communities really converge to make these models more powerful.

AI assessment note: “They are two different things.”

Q Yeah. So what, what are big enterprises doing currently? Are we still in a stage where they, they brought in the consultants, they build the, they build the center of excellence? I mean, what are they doing and what would you recommend they do to move, um, as fast as they need to on General ABI?

A So one thing that they've already done I see is I think computer infrastructure has been laid out more in place. I think budgets are there. Um, I think the next thing around organizational structure that's extremely important that I think people don't fully realize, which is relating, um, the technical piece of building out a successful model with the development piece of this model. And what I mean by that is, um, generative AI is famously, famously very hard to evaluate. We like have no idea what's good, better, best, unless someone who's an expert in understanding that That use case can tell you that, right? Like I can't tell you a medical, uh, you know, generative LLM, generative AI model, um, is actually that good. Cause I'm not a doctor, right? I can tell you when it's at like toddler teenager stage, but by the time it's getting an MD, I'm, I'm out, I'm not there. Um, so being able to have those people sit closely with the development teams actually accelerates, uh, those use cases much more quickly. And that becomes an organizational problem that becomes a very, very big Difference I see across some enterprises where those teams are closer together, so those use cases can get out much more quickly. And then other enterprises where those are much more disjoint today, so they need to reorg to be able to actually get those closer together in order to deliver those applicati…

AI assessment note: “So one thing that they've already done I see is I think computer infrastructure”

Q Uh, while we're at it, um, talking about, uh, definitions, you mentioned pre-training. What, what is pre-training versus training?

A So pre-training is a form of training, actually, and I think people often think of those similarly today for LLMs, but essentially it's getting the LLM to read the entire internet one word at a time or one token at a time, um, and it's just auto-completing the internet, right? Um, and so that's what it's tasked to do. Uh, it doesn't know really how to answer questions, so if you ask it, what's the capital of France? It's gonna respond, what's the capital of Spain? Because it thinks it's in maybe a survey context. Uh, and so after pre-training, uh, most foundation model companies, uh, for example, like Meta with Llama and, um, uh, OpenAI and Anthropic, they, they will do something called instruction fine-tuning, which is teach the model how to, Um, follow instructions so that when you ask what's the capital of France, it will say Paris. Uh, and so getting it there so that it's chatting with us. And then the next step, uh, we often take, uh, customers is memory tuning because that's very, it's not a general skill for the model. It's more so specific to your data. So to be able to embed facts of your data into the model, so memory tune the model so that it can recall those facts almost deterministically within its probabilistic context. Um, so That's, that's kind of the last step there. And then, of course, with inference, you can apply RAG and prompt engineering on top of that mo…

AI assessment note: “pre-training is a form of training, actually, and I think people often think”

Q We alluded to it. Uh, it feels like, um, you all, uh, had a major breakthrough in terms of, uh, research. Yeah. What is it? What does, uh, anyone need it and how does it work?

A Yes. Um, so maybe stepping back, these models are, you know, pretty good at everything, perfect at nothing. Um, and it's because they're trying to, uh, you know, they're reading the whole internet. They're trying to, They're, they're trying to just reduce average error, average error over everything, but average error means you're not actually perfect at those facts. Um, and it turns out we actually want something closer to deterministic on those facts. Um, and so how do we mix what is probabilistic and really good at understanding similarities where we want hi and hello to mean kind of the same thing. Um, but we also want those facts to be absolutely perfect, where if you get You know, your birthday off by a day or two. That's actually a, that's a major issue. If you get your, uh, revenue number off by zero, that's a major issue. Um, but from the model's perspective, it's, it's not. Um, so, uh, how do you make sure those, uh, for those facts, those specific facts, your business or otherwise, um, there's no alternative for the model. Um, and so memory tuning does that is essentially, uh, we say, you know, brings the loss to zero. So it brings the loss to zero on, um, All of these facts so that there is no alternative. It cannot even pick something that is similar. So something slightly correct, um, is incorrect as opposed to being, oh, kind of similar. Let me maybe consider tha…

AI assessment note: “memory tuning is a technique to do that”

Q As I was prepping for this, um, I also read about, um, your PEFT, uh, framework. I don't know if that's the right term, framework, but PEFT, which stands for parameter efficient fine tuning, uh, and then it was a concept of model switching. So what, what, what does that mean?

A Yes. What does model switching mean? So let's say you have a single server, like a single mode with eight GPUs or something like that, or you just have, let's say a single GPU. Let's make this super easy. Um, and you have a model that can run on that server, right? On that GPU. Um, great. What if you have, you know, let's say you have a thousand different customers and you fine tuned a model for every single one of those customers. Okay. You would need a thousand GPUs to serve each of them on, and that's like doing it the normal way. You would need a thousand GPUs. That is really expensive. Um, and also they're not super available right now, unless you want to go the AMD route with us. Um, so that is not something we recommend necessarily to, uh, scale it out to a thousand, um, GPUs per customer. Um, so what can you do? What does model switching mean? Um, if you wanted to just use one GPU and switch across Across those a thousand customers. So let's say a thousand customers are hitting, uh, their models and you're just switching it on the GPUs. You're changing which models loaded up on the GPU. Um, uh, the estimate is about, you know, three months of switching time, just pure sheer switching time. It would take three months to serve all thousand customers. And I don't think that's quite real time, um, inference, uh, from my vantage point, you ask a model something three months …

AI assessment note: “that time to switch across a thousand models is three milliseconds”

Q Uh, maybe a quick word on that, on that exciting partnership with, um, with AMD you alluded up front, but that, that's actually a, a product, like a package, I think you call it the Superstation, is that, is that correct?

A Yes. Um, our LLM Superstation with AMD. So yes, our secret's out. Uh, we've actually, um, in terms of our hosted service, the Lamini hosted service over the past year has been running on AMD, uh, GPUs only. We haven't been running on NVIDIA chips. Um, of course, for our customers, uh, we have, um, you know, our customers, when they're in their VPCs, we run the Lamini software package on top of their NVIDIA GPUs, um, almost exclusively for that. Um, and some of our customers now have both. Because they have now purchased, uh, essentially this, um, uh, these LM super stations that include, you know, Lamini pre-installed on them with, um, AMD compute. Um, and the reason for this is because of the compute shortage. I think without the compute shortage, this would, it would not be exactly, um, uh, a thing we, we necessarily be doing very explicitly, uh, but with a compute shortage needing, GPUs that are powerful enough to be able to run some of these models, you know, Lama, seventy billion, but even Lama, thirteen billion and be able to fine tune it is, is, you know, it's compute intensive. So you need that compute. Um, and as a result to unblock a lot of our customers who can't get them on a tier one cloud. And by the way, that includes, um, fortune, 500 and very large enterprises who can't do that. Um, so, you know, we've offered this as one solution to doing that. And we offer it…

AI assessment note: “Yes. Um, our LLM Superstation with AMD. So yes, our secret's out.”

Q talked, you were very focused on fine tuning, which is, uh, you know, part of the training world. Yeah. Uh, and you now added, like, the other side, which is inference. And maybe, you know, I, I said upfront, like, we'll, we'll define a bunch of things. So, like, for, you know, uh, anyone that's not super Uh, in the weeds of, of, uh, of AI, what is inference? Hmm.

A Inference is when you run the model. When you send chat to PT a text, uh, you're running inference. So, um, oftentimes we are, we're running inference of the model. Um, and, uh, yeah, we offer inference now. Um, we actually, it always been bundled. I almost view the future of, uh, a world where we will get to, you know, continual fine tuning, continual inference, and continual kind of requiring that integration, that bridge between the two, because they will both be Happening at a very fast speeds. Um, but essentially, uh, my view of inference today, you know, from a market perspective and, you know, startup market perspective is that, um, it's a race to the bottom today for cost. I don't think that's a controversial statement at all. Um, I think people know that it's getting priced lower and lower. Um, so we just offered forty million tokens for free. And, um, I think, you know, in a sense we can, as a startup, um, be able to say, Hey, actually we, We are differentiations in tuning the model, so we can actually just offer inference completely free. Um, and so this is essentially a way for us to initially get customers, um, and initially get them started, and then future-proof them as well. Enable them the flexibility to be able to then create, you know, use these models, but also improve them, um, over time for those future use cases if they are only on inference today.

AI assessment note: “Inference is when you run the model. When you send chat to PT a text”

Q Uh, why are you, what's the secret sauce?

A Yes. Uh, so a huge part of this is around being able to fully utilize GPUs. I think what we found was a lot of the open source packages out there, while incredible, and I root for those teams every day, Uh, they're not necessarily optimized in the multi-GPU case, um, multi-GPU scenario. So if you're using more than two GPUs, which nearly everyone I know is, um, then, uh, fully utilizing the GPU is actually incredibly important to eking out, um, eking out every bit of it you can. This is a very expensive piece of hardware you just bought. Um, if you're only using a small percentage of it, um, and you're, and you're running open source, it actually is more cost effective to run something, To pay us to install the inference, right? And to run that at high performance so that it is fully utilizing your GPUs, um, at scale for more than two GPUs. Um, so I actually find that dichotomy pretty interesting. Uh, so throughput is, you can think of it as batch latency. It's essentially how, how many queries you can put in. It's slightly separate from, from latency itself of a single query. Um, but that again is, I think a way companies are differentiating today as well. Um, and, and on inference, actually, I think the final thing that people are highly differentiated on is, um, uh, close to something, you know, with function calling. So being able to output structure, um, is really importan…

AI assessment note: “a huge part of this is around being able to fully utilize GPUs”

Q very much in the, you know, probable trenches of, uh, of AI, specifically enterprise AI day in and day out. What's the current mood in, you know, July, 20, 24, as we record this, it's sort of, It feels like the, the AI hype is slowing down a little bit. What are you seeing in the markets? Um, how are customers responding? What are they doing? What are you seeing?

A Yeah, it turns out having a non-negative, uh, margin is, is an interesting thing, isn't it? Who knew? Um, okay, so a few things. I think one is from the customer perspective, from enterprises, we sell enterprises. I, I think what's going on there has been extremely, uh, exciting, but also I think there's been met with, you know, these enterprises are actually starting to structure their organizations, having centers of excellences, um, who are tackling this problem, being able to onboard different Products internally. Um, and I would say the sentiment is, hey, we've tackled the shallow use cases. We've been able to put some of those in production, but now we're, we're thinking about what's next. Is this, is this technology really only going to help me compose email? Um, or is it going to do something a little bit more? Um, is it going to help me leapfrog my industry or is something in between that? What's, what's, what are the steps I need to take to get there? And so I'm starting to see deeper use cases. I'm starting to see people tackle those deeper use cases across their both data science and Um, engineering and even infrastructure teams.

AI assessment note: “we've tackled the shallow use cases. We've been able to put some of those”

Q Could you map the Uh, the inference market for us, like, who, who does what? There seems to be a number of different companies doing different things, something you have together, something you have model, this open source project, like BLM, who, who does what?

A Yeah, I'll, I'll map that out and kind of focus on, like, what are the key differentiators? Uh, so one key differentiator that I think we often see on benchmarks is around latency. Um, so latency is how fast the model responds to a single query. And, uh, you know, that makes it so things are real time when you chat with ChatGPT and that can move faster. Um, Unfortunately, I think what people don't realize is that the way to get better latency, like significantly better latency, is actually in the hardware. And that's why we see Grok, G-R-O-Q, um, be able to exceed all these GPU-based, um, uh, inference platforms significantly, by a significant margin. Um, and I think I saw in a VentureBeat article about 280,000 users now using that. Um, so props to them for, for getting that there. So I think for latency, if you want to compete with speed of Um, speed of the LLM's response to you, um, for a single response, then that is, that is, you need to compete at a hardware level. Um, the next thing is, uh, around security. I think, uh, security is interesting because are you running it on a cloud, right? Or that someone else is managing for you, or are you running it on your own GPUs? Whether that be your own GPUs in a VPC or your own GPUs, um, on premise, right? Um, so I think that there's a distinction there of What level of comfort do you have in sending data over, um, to another part…

AI assessment note: “Yeah, I'll, I'll map that out and kind of focus on, like, what are the key differentiators?”

Q just building some amazing stuff. Um, I just would put myself in the shoes of a customer, uh, in just like the level. Precision and sophistication and all the things, um, obviously it's exciting, but maybe daunting. Like, do you find yourself, uh, sort of doing a lot of hand holding, uh, in, uh, I guess, what is your experience to date and interacting with customers in the real world?

A Yeah, I, I would say, um, I'm actually impressed with the rate of learning that customers have had over the past year. We started before ChatGPT, and it was very preliminary. You know, like, customers didn't even know the word. Obviously, generative AI was not a thing. Large language models wasn't even a thing, so just a very different ecosystem. That was obviously a huge amount of knowledge transfer that we would have to do for a customer, and it also narrowed who our customer, initial customers, could be. Uh, today, customers are learning very fast, um, and we see customers at varying levels of the maturity curve, but they are, they are learning very fast, so they'll come back to us after a month or something and be like, we're ready. Um, uh, so I, I'm seeing that as really fast, and of course, I've been trying to contribute to that as well through the courses, like fine-tuning LLMs, which now has reached many customers who have now tried all those techniques and best practices, and I'm really happy about that. Oh, it's a, it's a scalable way of doing it. Um, but of course we do have change management as a thing. Uh, it is still real, realistically a thing, um, at different levels of a company. Um, and realistically having, um, you know, experience and also been around, you know, our team has been around a lot of these deployments in the past. It's that It, it isn't easy, you…

AI assessment note: “I'm actually impressed with the rate of learning that customers have had over the past year.”

Q Okay, fantastic. So, um, you cannot resist again to make this, um, interesting to a broad group of people with your, you know, academic hat on, uh, let's do 45 second definitions of some of those terms. So, uh, let's say, uh, what is, what is Laura?

A Oh, Laura. Okay, so when you tune the, uh, tune a model, so you have this giant model, right? And if you want to tune all of the weights of the model, which is traditional fine tuning, very expensive, very heavy to, like, change, make, do all those, you know, multiplications, additions, etc. Um, so it's very, very expensive. Um, Laura is a efficiency technique to make fine tuning extremely efficient. Um, it's part of the category class of Parameter efficient, fine tuning, peft. And instead of training all those, all those weights, all those parameters, um, you can just, you can tune only a subset of them, um, and sorry, not a subset, an external set of, of weights, um, instead. And then at inference time, fuse those weights using, I'll just say math, um, back into the model so that it Is the same latency. So it's not just external, like extra weights and it, you know, the model has to like compute all of that. When you ask it a question, you fuse it back into the model. So it is the same latency when it comes back to you with a response. Um, so it's a very, it's an incredible technique. Um, I think for something like a GPD three level model, it's a 10,000 X speed up in efficiency while losing nearly not much, uh, at all in accuracy. Um, and for enterprise use cases, that's really not the bottleneck. Like real bottleneck and accuracy. So we see it as just a performance improveme…

AI assessment note: “Laura is a efficiency technique to make fine tuning extremely efficient.”

Q Uh, where, where does that fit in the Lamini vision and, uh, and what are your, your general thoughts on the world of agents?

A Yeah, I think spicy topic. So I think, uh, so we support a lot of agent use cases today. Um, and obviously the LLM is behind, uh, running those agents. I, you know, the case studies on a SQL agent, right, able to ping a database, um, with the SQL it's generated and be able to respond back to the user. Um, my view of agents is that it's kind of just like, it's, it's kind of object, like object oriented programming, um, where it's, um, taking, uh, A different center. Instead of looking at each LLM, each LLM call, it's saying, okay, here's something that is almost like a, something I understand, like an object or a person, um, and being able to say, okay, it can do all of these tasks, and all those tasks are LLM calls. Um, so I kind of think of an agent as, um, something that does more than one LLM call and is, like, thinking about it from this, you know, center of gravity of, like, How person like can I get this so that I understand it in my worldview? Um, it's interesting because historically, uh, AI researchers, we haven't really thought about it from the lens of agents. I think, um, that really has emerged from the community, um, of both developers and, um, cause I think code agents became incredibly, um, uh, interesting and hyped up, um, just a little bit. And I think, um, from like a marketing perspective, I actually think it's, you know, And, um, an explainability perspecti…

AI assessment note: “we support a lot of agent use cases today. Um, and obviously the LLM”

Q Yeah, no, no, no. Well, hopefully we can do this on a, on a regular basis. I very much enjoy those conversations. Uh, so, uh, joke aside, what, what is next for That was a joke, man. All right. While it's on record now, you have a permanent invitation to come back. So what is next for Lamina?

A Um, so what is next? Uh, I think, um, we want to drop the speed of doing memory tuning significantly, um, because we've seen huge value in the market for this technology, obviously with reducing hallucinations being a major gap in production use cases. Um, but dropping the speed, I think people sometimes think, oh, performance, whatever. Um, Serious drops in speed and in performance, uh, result in the ability for completely new interfaces and tasks. Um, so if instead of hours, it takes minutes, if it takes seconds, um, I believe in a future where we're continuously fine tuning these models, um, where it's as easy as prompt engineering and, um, you know, these models continually improve. It actually baffles me that when we do text chat GPT and we give it all this feedback in my conversations with it, uh, that It's not actually adjusting the weights because it is quite difficult to do that. Um, and so to make that significantly easier and then also faster, those are two important pieces of, um, of, of bringing that future out. And I think that is the full capability of these models because today when people are running RAG or prompt engineering, um, those are, those are search. That's not AI. Um, it's like keeping the AI frozen and fixed. Um, and so I'm curious about a world where The AI is able to continually evolve with us, just like we learn. It's not like our brains are not p…

AI assessment note: “we want to drop the speed of doing memory tuning significantly”

Q we all expected as enterprise users or business users to be like sitting down in front of the model and keep giving it some thumbs up or thumbs down? Um, so, but, but it, it sounds like, uh, that, that could be the case, but like most importantly, just to play it back, uh, the, the future is, um, actual models doing this for. Uh, on replacing humans for this.

A Yes. Uh, it is actually, um, I think there are models, but then where humans come in, isn't this weird outsource task force in the Philippines, um, doing labeling. It's actually just your users using your product. And so there's ways to bake this into a product, a user experience that I think it's actually quite standard. Um, when I think of a product like Amplitude, for example, it's able to look at, you know, again, very just standard software engineering, standard products, Be able to see how users are engaging with a product and be able to, um, segment those users and also understand, like, all the clicks that are going in into, um, what a user is doing, and then you're able to do analysis on, hey, this, these are users that successfully went down this funnel, et cetera, and I think the same thing can apply here. The absolute same thing can apply here. Um, of course, I think people are a little bit, you know, worried with language models. They don't even grasp it, but I think that's the future. The future is analyzing, hey, based on, In the same vein based on how users are interacting with this product. Maybe it's, you know, editing something it generated. Maybe it's, you know, replacing it. Maybe it's accepting it and like sending that email, for example, if it generated an email, those are all user engagements that can then be fed in via RLHF, right? So that is, that's my…

AI assessment note: “where humans come in, isn't this weird outsource task force... It's actually just your users”

Q this, um, uh, you know, a, a, a discussion around, uh, smaller models and open source models versus commercial, like the very large, uh, models. Uh, I mean, it, it sounds, uh, without, uh, Uh, you know, my words, not, not yours, uh, but that you, you very much, um, in the camp of like small models and open source for discrete use cases. Is that a, is that fair?

A I think it depends on how you define small. Um, so, uh, we can train up to a hundred billion parameters. Um, so that's, I think sometimes in the large range now. Um, and, uh, what we, I mean, but what we encourage is, is, um, actually starting with some of the larger models that are just, you know, start at a better checkpoint and then moving smaller to optimize for efficiency. Um, and so what that usually means a common path is not like your own custom hundred billion parameter model, which of course, you know, happy to take that in is starting off with llama to seventy billion and starting from there. Um, and of course our, our customers are, you know, playing with all the different llama series. And I think to determine also other, other constraints are around compute. So how much you want to be spending on compute also matters. Um, and then how much data you have. Uh, so the amount of data that you have significantly, um, influences which model you should choose, uh, because if you don't have a lot of data, um, sometimes that might mean, you know, a smaller model can be better to, like, fully utilize all of that if you're doing pretty heavy, um, training on it, and I would call that, like, the domain adaptation training, like, the very heavy training on it, but if you're doing something very lightweight on top of something, Then actually starting with a bigger model where y…

AI assessment note: “I think it depends on how you define small.”