The Exchanges

Q And maybe to put, um, uh, terms around it, can you quickly define for us supervised versus unsupervised versus self-supervised?

A Yeah. Uh, so, so the approach of like, you have been like, um, uh, purposefully like curating the labels for the models trained from like fraud detection. So like, for example, you, you have to, uh, just change it down. Maybe like, here's the transactions, here's the label. And the label can be like fraud or not fraud or like spam detection. Then the, The data is an email and then the label is like spam or not spam. So, so you have to, to curating, like creating those labels, the process of manual, um, like any, you teach the model, like you like learn from those labels. So that process is like, um, it's like supervision. So the other spectrum is like unsupervised. It's like, you don't need to tell the model, like, uh, the, the, the, the, the labels and the model figure it out for clustering. So if you throw in a lot of, like, uh, articles to the model and say, hey, try to, like, group this into, like, five groups. So you don't need to tell the models, like, okay, this group is technology or something like that, right? Like, you just, like, The model can, can, can do it. Like a lot of clustering algorithms are like unsupervised. Um, language modeling is, is somewhere in the middle. It's like, uh, it's self-supervised. And I, and the reason is that it's still like learned from some labels. Like, first of all, like the next word in a sequence is a label that you would need to lea…

AI assessment note: “language modeling is, is somewhere in the middle. It's like, uh, it's self-supervised.”

Q very opinionated about a certain way of doing things or, or, or when you teach it, uh, do you present it as this is like a, you know, series of, of, of, of tools and as long as you have, uh, an idea about what you want to do, like everything can work. I guess what, what are the, uh, the, the key opinions you have on machine learning design?

A So, uh, so the main idea is you go backward from the, the problems. So I think a lot of approach machine is saying it's like, you start with the solutions and it tries to have like five problems when machine can I be applied. So, and it's like, so tend to be like, oh, Hey, this is fancy model coming out like bird or whatsoever. And then let's try to like, see, like, um, let's try to run it and see like whether it help us. And I think it's just like, it's, it's very interesting in R and D, but I think it's just like a wrong approach when you try to slow actual problems. So I was trying to encourage my students to like, look at the problems. Like what problems is this? Like, what is the easiest, simplest solutions? And it doesn't have to be machine learning. It can be like non-machine learning or, or if it's machine learning, it can be a very simple model. Uh, and then from there you can, so at least you have some baseline and then you can touch on more complex system models for, uh, for the solutions. So it's a lot about like best practices. Um, yeah.

AI assessment note: “the main idea is you go backward from the, the problems.”

Q So the Generative AI stack, that's the foundation of, of Generative AI. Um, what, what are the different components that people should know?

A Um, so, so when you look into, like, building applications, I would think about, like, maybe a developmental process, and maybe the stack should be, like, evolve to, like, address, like, your needs, right? So, so when you start with applications, maybe you start building, like, thinking about, like, um, maybe you start with, like, a testing out the models. So you might want to, like, do some problem engineering, and see how far you can get good problem engineering, or maybe you need to curate some evaluation metrics. You definitely need to evaluate, like, To design some evaluation metrics. So, so I think this is like the, the application development layer, right? Like, uh, with problem engineering, uh, maybe it will be like how to like enforce like structure output, uh, security guardrails, um, maybe like with, uh, definitely with, with evaluations. And then like, after you do it, so we can application layer, and then we, um, we go and then we, we max out that performance there. And then we're going to, Hey, maybe we need to change the model, right? Like maybe we We, we need to like fight to the model. We need to make the model smaller and make it faster, like inference optimizations. So like that layer, when you actually make some changes to the models itself. So it's like the model development, like fight tuning layer. Um, and then like, after that, uh, I think you go into li…

AI assessment note: “this is like the, the application development layer... model development... infrastructure”

Q That's what you called, uh, auto-regressive models. Is that right?

A Uh, yeah. So, so I think I said that auto-regressive, um, is definitely one type of language models. So I think that's the idea of like all language models is that you encode like statistical information. Um, and the concept actually not new, like, uh, I think, like, uh, people employ that to, to, to, like, decode, like, to break code during World War II, right? Like, people, like, using that, like, for, for games, and it's very, very interesting. Um, so, so autoregressive, like you mentioned, is, like, should predict what comes next, whereas, like, the math one is, like, you, you can have a context from both before and after, and predict what is the middle. And, and both, um, what, what kind of language modeling tasks actually, um, can, The data is, like, abundant. Like, if you have some tasks like machine translations, you would have to, like, curate, like, here is the original sentence, and here's the translations, and it can be quite painful to curate that. But for language modeling, uh, because you can just have any natural text, like, online, like, it's, like, so much of it online. You can just use it. And also, like, not just, like, a natural text. You can use, like, programming languages. I can use, like, code bases. Um, and, and I think, like, that is just, like, that's the nature of it. It's like, you don't need to curate, like, labels, like, reference data, so that, …

AI assessment note: “Uh, yeah. So, so I think I said that auto-regressive”

Q You mentioned a term, um, called sampling in, in the book that, um, uh, is very interesting, and you mentioned very, uh, very important in terms of understanding how those models behave. Can you Uh, maybe define that for us?

A Um, sampling is really fascinating. Um, I, I think it's, it's, it's actually writing section sampling is like one of those that bring me the most joy because I really like it. Um, um, so, and I feel like the topic is really underrated. Um, so sampling is a process of a language model, um, picks a possible, like pick one output out so many possible outputs, right? So, so we talk about the language model, like encodes statistical information about language. So let's say I said, it's like, um, the answer to this question is. 70%, yes. 30%, maybe a 20% no and a 10% maybe. Right? So, so the model is like models and look at all these possibilities like, hmm, what should I pick next? So maybe 70% of the types in the pick yes and like 10% is a big no and a 10% is a big maybe. So, so that is a sampling process. Um, and, and, um, and the language model, right, you, you don't just, it doesn't just sample, like, each response is not just one token or, like, one word, right? It has to respond, like, sample, like, over and over and over again. So sampling refers to, like, um, different techniques so that you, different strategies to, like, notch the model to pick the, the, the, the output that is more valuable to you. Um, so, so let's say just like, um, for example, like one, one thing people do is like temperature. So, so, so you can nash the model to like pick more frequent tokens. For exa…

AI assessment note: “sampling is a process of a language model, um, picks a possible, like pick one output”

Q Still on the topic of evaluation, an interesting tidbit you talk about is the concept of AI as a judge. So some, somebody or something needs to evaluate the, the, the system. It could be a human, it could be AI. What, what are your thoughts on the, the pros and cons of AI as a judge?

A Yeah, um, I do think it's like AI as a judge is a very promising approach. Um, so, so I do think that, um, I think when AI first, uh, when Chelsea first came out, AI as a judge was, it was just like, AI is not reliable enough as, uh, to be entrusted with a crucial task. But then nowadays you talk to like teams, I think like most teams have like some variations of AI as a judge going on. Um, so, so AI as a judge is pretty interesting. The idea is that like you have a AI, um, Evaluating the outputs of like other AI, and it's especially useful in productions. The idea is that like, um, let's say like the model, you use a model to generate a response. A lot of people were like, oh my God, what if the response is like not safe? What if the response is like crazy? What if it's like saying something like, get me Sue? So, so maybe you can have another model. It's like, just, you double check this one and give a score and send back. Um, so, so I guess Josh is like, um, pretty, it has been like, Able to show to work like very, pretty strongly correlated with like human, uh, human judgment. Um, and the tricky thing about ASS judge is that, um, ASS judge is not as, how to say, exact as, uh, it's not, it's not, um, subjective as other metrics like F-one score or like, uh, so, so what that means is that like, when, when, when somebody says F-one score, you, you know that You know what that m…

AI assessment note: “pretty strongly correlated with like human, uh, human judgment”

Q you make the point that, uh, while AI adoption seems new, it's built upon techniques that have been around for a while, like language models, some of which came in the 19 fifties and then retrieval techniques. But, um, At the same time, it feels like a new field. So, what is new about AI engineering, and, uh, how is that different from more traditional machine learning and MLOps techniques?

A Yeah, um, I, I think that, that's a great question, and I get asked that question a lot, and was like, okay, what is engineering? Is it another marketing term? How is it different from my traditional MLOps engineering? So, um, there are a lot of overlap between these two, like, roles, and I think a lot of companies, like, even, like, People with the same title, like, and have very different functionalities. So I think like any definition is like a little bit like fuzzy and like really depends on like where you work and what you're working on. Uh, but in, in general, I think of like machine engineering is when you like had to build the models yourself, like before, like before the availability of like large language models or foundation models that anyone can access. If you wanted to build, um, ML applications, you could need to build the models. Yourself. And only a few organizations could do that. But nowadays, like anyone, like anyone who wants to leverage AI to build applications can like just leverage one of those amazing available models to do so. And it just makes it so much more accessible. And another thing is that like, um, before, um, I had thought that, um, a small improvement of like AI capabilities Could lead to a small, like, increase in the number of available applications. So, like, we know for a long time, we have known that if we put more data and more compute…

AI assessment note: “machine engineering is when you like had to build the models yourself”

Q So what, what are the key concepts, um, around evaluation? So you, you mentioned, uh, Couple of those terms, uh, like entropy and perplexity. What are the criteria? What are the methods? What should people know about?

A Yeah, um, so, um, so you mentioned like entropy and perplexity. So those are really fascinating concepts, and the action is it guides the development of language models, but because most people today are not going to build a language model from scratch, you might really do it for fun, but not going to be as a scale Where you can compute OpenAI. Um, so, so, but, so, but, so, so I think like, uh, entropy and complexity is useful to know, but pretty not what you are going to use day to day to evaluate your applications. Um, so, so I can talk about like entropy and complexity, I think they're like really, really, really cool concept. Um, so, so on, on, on things I want to mention about that is just like, Um, we want, like, entropy to be, like, lower. So, so, so, like, to make things, like, basically more predictable. So, so, so, for example, like, if the model is getting really good at predicting, like, the next token, so, so that means, like, now the Trinity that becomes, like, that language becomes, like, more predictable to the model, right? So the entropy is not below. Uh, and over time, people find out, it's like, hey, if I could just Decrease entropy. Somehow users are happier, like all the users using the applications and become happy. And then the question is like, how far can I go? Like how low can the entropy go? Right? Because like, absolutely it can't go to like zero. I…

AI assessment note: “if I could just Decrease entropy. Somehow users are happier”

Q And, um, what have you, uh, learned about the evolution of the landscape over time?

A Um, it's very interesting. So I was looking at it and I think it was trying to, to look at the year. It was a tool was, uh, made the first comment if it was, uh, open source or if it was a company, it was looking at when it was incorporated. And I was, uh, so I was trying to divide the tools into different categories based on the problem it was trying to tackle. Um, and I think there was, um, this like a pre, uh, deep learning phase, like before AlexNet in Traditional older like framework with a lot of cool things like a decent tree and stuff like that. And then after deep learning, there was this phase, this explosions of deep learning framework. Um, and, and, and then from my, uh, I think, um, I think Google has, um, um, okay. My phone is just like, Woke up because when I say Google, my friends say, hey, I'm young. So, um, so, so whenever you, um, so, so after, from, in 2016, I think Google has this article about how to use deep learning for, for Google Translate. Anything was one of the, or one of the first, um, one of the first use case of like deep learning in productions. And I think from then there was an explosion of, of companies like trying to use deep learning in their, in their, uh, in their products. And we've seen a lot more tools around like bringing machine to production. So less framework. So I think by pretty, in, in, in, in, in, it's pretty much the competiti…

AI assessment note: “we've seen a lot more tools around like bringing machine to production.”

Q Sounds good. Uh, and, uh, let's see one more. Um, Uh, what are some applications that are leveraging online learning today? EG recommendation systems for e-commerce and how is this implemented on device or cloud?

A Oh, so I think it's a, it's a great question. So I was applications of, uh, leverage online learning. Um, so I think that one of the biggest one is, is, uh, like the easiest one to see is definitely, um, uh, recommendations. Uh, so TikTok is one of the, like, um, the biggest example for this, uh, like you, like, People attentions online and what, what we are interested in online, like changes, like second to second first reasons, like you went online thinking you could, you could watch a lecture on machine learning and then you read some news about like octopus punching a fish and like, okay, now I want to squash a video of octopus punching a fish. So it's just like our attention online to change very quickly. Yeah. It happened to me all the time. Um, so, so, so, so that's, that's, that's one of the use case when you want to like learn and adapt to user. Um, preference very quickly and then make predictions is just, just what they want to run as an online recommendation system for online content or so e-commerce. Um, another use case, I think it's like heavily underexplored and I don't see a lot of companies doing it except for a few, like I think, um, except for very few, few big companies. So it's a customer service support. So right now people are trying to like reduce, like, um, to, to make customer support more effective. So, so, so when users support a customer ticket, yo…

AI assessment note: “one of the biggest one is, is, uh, like the easiest one to see is definitely, um, uh, recommendations.”

Q all that says, as teams rush to adopt AI, many quickly realize that the biggest hurdle to bringing AI applications to reality is evaluation. For some applications, figure out evaluation can take up the majority of the development effort. Um, so you mentioned some of the specific challenges, uh, of, um, of, uh, why is it so hard to evaluate? How does, uh, one, uh, think about evaluating those models?

A Yeah, um, evaluation is hard. So, I think, like, I realized this term, it's like what I call, um, evaluation driven development. So this comes from engineering, the concept of, like, test driven, um, development. So the idea is that, like, um, you Develop, uh, applications that you can evaluate. Right. So, so, um, even though it's like, I think I see a lot of people who I said about like the latest marketing buzzwords, I think it's like one thing I've realized from working with a lot of tech executives is that they actually really smart. And I think like a lot, like surprise, you become the SVP of this giant corporations because they were pretty smart. Um, but, but yeah, so, um, I think a lot of businesses, business decisions are still made based on return on investment. So that's like, it's really hard for people. Should I say like double down on something if they can't say just like, Hey, this is making a real money for us. So it's not a surprise. It's not a coincidence that some of the most popular AI applications today are those that you can evaluate the output like pretty clearly. So for example, like recommended systems, like everyone has a regular system nowadays, and it's because like the regular system, it can tell like, How much money is bringing in by, like, whether it's increasing, like, say, like, click through rate or, like, purchase through rate, right? So, like,…

AI assessment note: “I call, um, evaluation driven development. So this comes from engineering”

Q And still on the topic of scale, uh, why does it matter how big a model is in terms of, you know, millions or billions of parameters? What difference does it make?

A Uh, that is very interesting question as well. I love how you're asking is this very deep philosophical questions. Uh, and I feel like I really need a whiteboard, uh, to explain all of this. Uh, so people, if you don't understand me, trust me, my writing is better than my speaking. So like why, what does it matter is as a model should have a lot of parameters. So parameters, like there's a numbers of parameters, usually like, um, Approximates like the, the models learning capabilities. So, so it's just more like, um, with, with more, so you can think of like having more parameters, more ways for the model should like learned, uh, information. So, so you can think, um, as I try, I'm trying really hard not to use the term like neurons, the brain having more. More, more synapses. You can learn more. It's just like, it's not that equivalent. Uh, but yeah, so, so basically you can think of like the number of parameters is more learning capabilities of, of the, uh, learning capacity with the model. So the more parameters allowing the model to learn more. Um, so, so there's actually a very interesting question that, that, um, is that like, why do larger models need more data to learn? Because the idea is that if the model has more capacity to learn, should it need less data? Should learn. Does that make sense? If someone is smarter, right, it should learn faster from less data. Yeah, …

AI assessment note: “the number of parameters is more learning capabilities of, of the, uh, learning capacity”

Q And there's a part, uh, in the prompt engineering section that I thought was, uh, particularly interesting around defensive, uh, prompt engineering, so defense around, uh, against jailbreaking, information extraction, that kind of thing. Can you talk about that and maybe what, what we have learned about, uh, uh, making those, um, AI systems, uh, resistant to that kind of those kind of attacks?

A So I do think that's a topic is getting increasingly important, especially as AI is being like first, um, it is being used for more like, um, high stack tasks, right? And a more complex task. And the second, um, is, is like, it's now AI has increasing access to more tools and it can make changes. Um, so, so I do things that like there are, um, so, so we, We do want AI, like, like users to be safe, like pretty, not just users, but also like developers of those models that nobody gets to. Um, so, so I think that, um, with, um, it's also like when one reasons that makes a lot of people like go to like proprietary models instead of open source model, because I say, if you use like a model, like developed by companies, like through the API, You, you kind of have some, like, um, these companies are responsible for, like, putting guardrails to make sure the model behaves safely, like, it doesn't say anything racist, sexist, somebody asks this question about, like, praising Hitler, maybe even say, no, I'm not gonna do it, right? Like, there's a lot of guardrails around this, but if you use, like, an open source model, and you de-price yourself, you're, like, responsible for it yourself. So, so, so I do think it's, like, of course, open source model developers try their best to make, to make the models, like, safe as well, right? But, but they also have less visibilities into how the op…

AI assessment note: “makes a lot of people like go to like proprietary models instead of open source”

Q being batch and the other one happening in almost real time, but, uh, is that, is that, um, is that also like a different algorithm, like a different stack? Is it, uh, like a completely different set of tools to be able to do that? Or, or, or, or does some of it come from the, you know, the, the, the very nature of the, um, of the media property?

A Um, so this is a good question. So, so they make questions of like, why are companies not doing that? So one question is, so, so I would talk to a lot of people and ask people like, why, why don't you do it? And they were like, why should we make predictions online? Because there's no point in doing it, like doing bad predictions and it's totally fine. So, so, so, so, um, here's one way of doing things and here's another way of doing things. And the other way of doing like my change the way you like, uh, set up so infrastructure. And it's just like, you have never tried before. And it's just like, and you've never tried before. So you don't know what boost performance boost you can get because you have nothing to compare with. So you might be very tempted to just like stay the other way. Um, so you're talking about like, what can be, um, what, what makes it hard? Uh, so like there's several reasons that make online predictions hard. I'm just talking about online predictions right now and not even online learning yet. So online prediction is still the easy part. Online learning is like really hard. Um, so, um, so there's several reasons I should do online.

AI assessment note: “So you're talking about like, what can be, um, what, what makes it hard?”