The Exchanges

Q of closed and open source? Do you think the most cutting edge models, the, the giant language models are going to be both? Or do you think like capital will eventually become such a large obstacle that it'll make, um, the private world more likely to drive progress forward? And I know you have plans in terms of how to offset that, but I just love to hear about those.

A The reality is we have more compute available to us than Microsoft or Google. So I have access to national supercomputers and I'm helping multiple nations build exascale computers. So to give you an example, we just got a seven million hour grant on Summit, one of the fastest supercomputers in the US. And like I said, we're building exascale computers that are literally the fastest in the world. Private companies don't have access to that infrastructure. Uh, cause governments, thanks to us are realizing that this is infrastructure of the future. So we have more compute access. We have more cooperation from the whole of academia than all of them do because their agreements tend to be commercial. There's no way that private enterprise can keep up with us. And our costs are zero as well. When you actually consider that, whereas they have to ramp up tens of billions of dollars of compute. So my take is that foundation models will all be open source for the deep learning phase. Cause we're actually got multiple phases now. The first stage is deep learning. That's creating of these large models and we will be the coordinator of the open source. The next stage is the reinforcement learning, the instruct models, flan palm or instruct GPT or others that requires very specified annotation. And that's something that private companies can excel in. The next stage beyond that is fine tuning…

AI assessment note: “my take is that foundation models will all be open source for the deep learning phase.”

Q That's really cool. And on the computational biology side, I guess there's a few different areas. There's things like protein folding, and then to your point, there's things like MedPalm. Are you thinking of playing a role in both of those types of models in terms of both the medical information? Yes.

A We will release an open MedPalm model. Well, med stable GPT. Um, and then protein folding, we are the, one of the key drivers of open fold right now. So we just released a paper on that, um, much faster ablations than alpha fold. We're doing as well, um, DNA diffusion, uh, for predicting the outcome of DNA sequences. We have BioLM around taking language models for chemical reactions. And that's an area that we will aggressively build because there's a lot of demand from the computational biology side for some level of standardization there. There have been initiatives like Melody and others looking at federated learning, but there is a misalignment of incentives in that space that I think we could come in and fix. And I think that's where we really view ourselves. How can you really align incentive structures and create a foundational element that brings people together? And I think that's where we are most valuable because private sector can't do it that well. Public sector can't do it that well. A mission oriented private company that has this broad base and all these areas could potentially.

AI assessment note: “We will release an open MedPalm model... and then protein folding, we are... key drivers”

Q like mature use case, uh, with instability, and as you said, media as an advantage, what does the future of media look like? And actually, if, even if we go back before that, you know, you're involved in, um, sort of Early ecosystem efforts, uh, with Luther and such. How did you even identify that this was an area of interest for you versus everything else going on across modalities?

A So, you know, I've always been interested in meaning, like, uh, semantic is even part of my email address, and, uh, that's my religious studies as well around epistemology and ethics, ironically. Um, the way that I viewed it is that the easiest way for us to communicate is what we're doing right now via words, right? And that's held constant, but now we can communicate via phones and podcasts or whatever, and it's nice. Writing was more difficult and the Gutenberg press made it easier, but visual communication is incredibly difficult. Be it a PowerPoint, which is visual communication or art, which is visual communication. And then you have video and things like that, which is just impossible. Now you have TikToks and others making it easier. And I saw this technology and I was like, if the pace of acceleration continues, visual communication becomes easy for everyone. Like my mom sending me memes every day, telling me to call more or kind of whatever. And I'm like, that's amazing because that creation will make humanity happier. Like you see art therapy, that's visual communication and it's the most effective form of therapy. What if you could give that to everyone? So there was that aspect to it, but then I saw movie creation and things like that. So my first job was actually organizing the British independent film awards and being a reviewer for the rain dance film festival. …

AI assessment note: “I saw this technology and I was like, if the pace of acceleration continues”

Q Oh, I was just asking about, uh, and I think Sarah asked the question around, um, you know, where do you think regulations should be applied and what are the, what would be positive outcomes of that versus negative outcomes?

A Yeah, so I think, you know, there should be, um, these elements around identification of AI, especially on advertising. I think that there should be regulation on very large models in particular. Um, European Union introducing a CE mark slash, uh, Generative AI restrictions where the creators are responsible for the outputs, I think is the wrong way, but there are other ones as well. Like I would call for opt-out mechanisms. And I think we're the only ones building those for datasets. Um, cause we're also building some of these datasets and trying to figure out attribution mechanisms for opt-in as well on the other side. Like right now, the only thing that is really kind of Checked is robots.txt, which is kind of thing on scraping, but I think, again, it's evolving so fast that people might be okay with scraping, but they may not be okay with this. Legally, it's fine, but then I think we should make this more and more inclusive as things go forward.

AI assessment note: “identification of AI, especially on advertising. I think that there should be regulation”

Q most obvious paths to performance progress is to just make models bigger, right? Scaling data, parameters, GPUs, which is very expensive. Um, and then in reaction, just to set the stage a little bit, there's been some efforts over the Uh, previous years to be more community driven and open and build alternatives like Luther. How did you start engaging in that and how did stability change the game here?

A Yeah. So when I was doing the COVID work, um, you know, we tried to get access to various models. In some cases, the companies blew up. Other cases, we weren't given access despite it being a high profile project. And so I started supporting Eleuthera AI as part of the second wave. Um, so, you know, Stella and Connor and others kind of led it, um, on the language model side, but really one of my main interests was the image model side. Uh, I have a Fantasia, so I can't visualize anything in my Brain, which is more common than people would think. In fact, a lot of the developers in this space have that, like we've got nothing in our brain.

AI assessment note: “And so I started supporting Eleuthera AI as part of the second wave.”

Q Yeah. Some of the other areas that you've talked about, I think in interesting ways is about how AI can be used to make our democracy more direct and digital, a little bit more about, um, you know, broader global impact. Could you, could you extrapolate a bit more there?

A Yeah. So I think, you know, if you look, you have to look at intelligence augmentation, right? Like information theory in classical Shannon ways, information is valuable in as much as it changes the state. And we've obviously seen political information become more and more influenced by like manipulation of stories and things like that. So the divide has been grown. What if we could create an AI that could translate between various things, make things easier to understand and make people more informed. I think that would be ideal with some of these national and public models and interfaces being provided to people. And then that can be very positive for democracy and allowing people to really understand the needs. Like you can already with ChatGPT, when you train it on nature of yourself, it can summarize for your perspective. You know, that's amazing thing, right? You can tell it to talk like a five-year-old or a six-year-old or an eight-year-old or a ten-year-old. Once it starts understanding Sarah and Elad, That'll be even better. And again, you don't need to send open students to do that. The open AI embeddings API is fantastic. But I think there'll be more and more of these services that allow there to be that filter layer between us and this mass of information on the internet that will be amazing. I think if we build the education systems and other things correctly as we…

AI assessment note: “What if we could create an AI that could translate between various things”