The Exchanges

Q Do you want to talk about, uh, mega kernels and Together Atlas for a minute?

A Together, uh, mega kernels, Together Atlas, these are both, um, projects along these lines. So let me dive into the mega kernels first. To understand this, uh, the, the first thing when we say kernels is we usually mean we are going to write a specialized GPU program for a single operation in a model. Um, you can think of a model as one of these train models is like ABC different operations in a row, and there'll be hundreds of these. And the way that we've been writing kernels for the whole history of, let's say call it NVIDIA hardware, is that you really specialize a single kernel for a single operation. With these mega kernels, we're doing something quite interesting, which is We can take the entire model, however many billions of parameters and put it into a single GPU kernel. Um, and with that, you can start to do a lot more fine grained optimization than you were able to do before. Uh, it actually starts to make the NVIDIA GPU look a little bit more like a Cerebris chip or look a little bit more like a Samba Nova chip in terms of the, the optimization that you're able to do. And this is really critical at inference time. So we're able to see two X, sometimes three X speed ups, um, over even highly optimized inference engines. Um, so we're working on bringing that to, um, uh, to, to really work in production, bring, bring it to fruition and use it across our whole stack. T…

AI assessment note: “these are both, um, projects along these lines. So let me dive into the mega kernels”

Q Okay. Fascinating. All right. Dan, what is your perspective on all of this?

A I really appreciated Tim's post because I think one, one thing that, that I really appreciated is that there's some, some AGI talk that if you just kind of like trace the exponential, at some point you get, you know, the, the thing that will eat up the, the universe or, or whatever, um, which I, I always found a little bit odd to, to think that way. I appreciate the, the thing in terms of the actual physical constraints, because Uh, you know, like Tim said, these are physical systems with physical inputs, um, and, uh, and actually doing physical computation. I think my perspective was that if you look at where the systems are today and you look at the, the models that we've trained, uh, we are just so far from being, from even using the last generation of hardware as, as efficiently as, as possible. So, and not to mention all the new hardware that that's being built out. So I think on the technical side, I'd say there, there are two major points I wanted to make In my post, which was one, if you look at the models that are kind of the, the really great ones, um, the, uh, that, that we know today, and I, in my blog post, I mostly talked about open source models because they talk a little bit more about how they train the resources behind it. Um, we don't have public figures behind, you know, how much open AI and, and Enthropic are using. Um, but if you look at the deep seek mode…

AI assessment note: “I think my perspective was that if you look at where the systems are today”

Q And Dan, you mentioned a second ago in reference to, uh, self-driving cars, that moment where, where things flip, switches, is turned on. Has that happened? With agents already. You talked about software singularity. Are we, are we at that moment for agents?

A Yeah, I think that so personally in, in my life, I'd say that moment was last June ish. So June, 20, 25 was the moment that, that it really flipped for me to give some context here. So what I do in my day job at Together AI is we write a lot of these GPU kernels. I don't know how popular, but the, in the, in the general ML zeitgeist, GPU kernels are thought of as kind of like the final boss of the thing that you learn how to program. They're very hard. They're very highly parallel. Uh, you don't write them like you have to write in C++, which is this old language that the old systems people use, you know, decades ago or whatever. They're not in Python, et cetera. When you're trying to hire for people who can write kernels, it's very hard. It's a very challenging skill set. It's certainly, you know, the, the, the, the, the tip of the spear in terms of, um, the, the programming strength. And last June we had this really interesting realization where we realized that cloud code, cursor agent, these agentic coding assistants were actually very good at writing these kernels. So, um, there was one week where I think I wrote like three or four different features that usually would have taken me a week each. And I wrote all of those in a single day. Um, and I was like, oh my God, this thing is making me five times more productive as a kernel expert. Um, I got my team on it. Now my team…

AI assessment note: “personally in, in my life, I'd say that moment was last June ish.”

Q If I understand this whole discussion correctly, all of this is about pre-training, right? And whether we can train a bigger model with, uh, you know, more data and more compute, but in conversations on this pod, a lot of the conversations have been about, uh, the importance of post-training and, uh, the, you know, building AI systems with pre-training plus, Plus RL. Where does that fit?

A That's a great question, and I think another piece that, that we, we didn't, you know, I don't think either of our blogs particularly hit on. One way I like to think about it is that pre-training is like the general strength training that you do in the gym. You go lift heavy weights, you improve your strength, improve your, your, your general ability, and then post-training is like the specific drills that you run to, um, to, to get a good, uh, at a particular task. So historically, the vast amount of compute has gone to pre-training. So just Building models that are more generally capable of doing many things, have a lot of knowledge, get to a point where maybe they, they have more knowledge than, than your average person. You know, I, I certainly don't know as much as, as ChatGPT for, for instance. And then the post training is both, how do you make it helpful? So, uh, you know, ChatGPT, you ask it to do something and then it actually listens to you, um, and, and tries to, tries its best to do it. But I think the other thing that, that we've started to see increasingly in post training is that you can start to post train Specific skills. So, uh, the model that's really good at helping you code uses a lot of the knowledge that you got from pre-training, but it's actually adapted to be particularly good for coding or, um, or the, or the, the model that's really good for legal w…

AI assessment note: “pre-training is like the general strength training... post-training is like the specific drills”

Q And then on your end, what have you learned or observed in terms of agents? What works? What currently doesn't work, but will work soon? How to manage them?

A I think there, there, there's two broad, um, things I, I, I've noticed for agents. So the first is, um, making the agents effective, uh, ends up being a lot like managing, um, junior folks on your team or, or at a company. So for example, the new intern who, who shows up on your team Um, you're not gonna go to the intern and say, hey, go fix our revenue for the year, double our revenue for the year, or something like that. Like, maybe you'll try that once, but you're, you're unlikely to, to see, um, the, the payoff from that. Um, instead of what you often do with junior folks is you say, hey, here's a first little task that you can do that in, to get to know this complicated code base. And here are the things that you might run into, um, because you, you've kind of done it before. When you give the agents that context, give them that ability to look at those things, then they can usually figure things out. The other bit is that when you have a new person on your team, you maybe won't give them access to all the production credentials and all the production database and all those things, but you're going to give them enough tools to, to be productive. Um, so sometimes there, there's this tension between, oh, I don't want my agent to go delete my everything in production. So I'm just going to Have it be hamstrung and watch every little thing it does. Whereas if you did that with …

AI assessment note: “making the agents effective, uh, ends up being a lot like managing, um, junior folks”

Q Dan, as a quick tangent, do you think we're evolving towards a multi-hardware, multi-chip kind of world based on what you see? You know, obviously there's been Grok and NVIDIA, there's, uh, Cerebras, there's like a bunch of, uh, sort of special ASICs companies coming up, uh, from, from your kind of like low level in the stack, uh, Vantage Point. What, what do you see?

A Yeah, that, that's a great question. So it's something that I spend quite a bit of time thinking about, um, more so I'd say on the lab side than, than necessarily, um, in, on, on the industry side. Um, although of course we're, we're paying close attention kind of on both sides. I think it's, it's at a really exciting time where, um, the Nvidia chips are really strong, really reliable. Uh, there's a lot of software support around them that, that, that has built around. We're starting to see the same things happen, for example, on AMD chips, um, with, with some, some of the research there. So on the lab side, we put out a, recently a library called Hip Kittens led by my great friend, Simran Arora. Um, and she was really looking at, okay, how can we take, what are the right software abstractions to, to program on these AMD GPUs? And it turns out they're not exactly the same as the NVIDIA GPU. So even, even two GPUs that have relatively Similar specs, certainly compared to Grok or Cerebrus or Samba Nova or one of these other chips. Um, even though they're relatively similar, they, they actually have pretty different software abstractions you need to use. Um, and I think more people are getting excited by that and, and investing time and, and energy into that. Um, we saw the Grok acquisition from Nvidia. A lot of people are excited about TPUs today. Uh, I think, uh, Cerebrus and Op…

AI assessment note: “certainly it's going to be a wave of things kind of coming forward”

Q the second part of this conversation, we'll talk about agents and coding agents and your thoughts there, because I want to make sure we cover that. AGI, uh, obviously it's a term that, uh, everybody uses, and I think we can all agree that nobody really knows what that means. But for, for purposes of this discussion, uh, what, uh, is a, uh, useful definition of AGI from your perspective?

A Sure. Yeah, I think so. One of the things that we kind of discussed back and forth in this set of blog posts is sort of what AGI means. For me, I think one of the things that I've been thinking about recently is that if you took where we are today, with the models that we have today, with the language models, and I think, you know, we'll probably talk about this a bit more with the, later with the agents. By almost any definition anyone could have written down, let's say five years ago or 10 years ago, certainly when, you know, Tim, you and I started our PhD. We basically have the vision of AGI that, that we had back then. We have things that can write code. They can write, you know, human text, even though, you know, maybe the, they, they use too many M dashes or something like that, but they can, uh, do, do these really, really amazing things. I think one of the things that I think about is at what level does this kind of become a new industrial revolution where you can, where this technology is really going to change a lot of what, The way that we do things today, the, the, and have a huge, you know, really, really great economic impact. In terms of software engineering, I feel like we're already there or almost there. Like, uh, there are, there are things that may be super specialized. I don't know if they're going to be able to write like the best Fortran and COBOL code co…

AI assessment note: “at what level does this kind of become a new industrial revolution”