The Exchanges

Q from, I don't know, A, as you said, just like having better ground truth data on the current state of the weather, B, um, more compute, as you said, has been running in supercomputers, so we get power, more and more powerful computers, we can just run more and more complicated Navier-Stokes equations, or C, additional tricks, Basically, that allow you to, like, do better predictions without adding more compute?

A Yeah, that's a, that's a great question. So I, I'll just admit, I don't know the answer to that. I think all three contribute. Um, so I can say on the first one, data, uh, yeah, we, we, like, have, there's, you know, better satellites that are flown, and there's, uh, more, uh, better systems for, you know, collecting balloon observations or these different sort of things. So we definitely are getting better data, and we know that that improves, uh, the quality of the forecast. Um, we're also getting better models. Um, That's definitely true as well. We're, we're, the, we're getting, you know, big, we're building bigger supercomputers. They can operate at finer resolution. Um, just, I think, uh, in the last, less than 10 years, uh, the ECMWF, which has the best weather forecast, uh, they increased the resolution, meaning that they had finer detail and space in their forecasts, and that allowed the forecast to be more accurate. So you see, both, like, adding just raw compute power, but also improving the quality of the models. And the approximations can also, uh, you know, has also made, I think, a pretty dramatic impact, and I think that sort of blurs into your third category of, like, other tricks. Um, I think in general you have, uh, you know, with, like, without getting into the details of how the numerical models work, you can kind of think about them as a backbone that's ma…

AI assessment note: “I think all three contribute. Um, so I can say on the first one, data”

Q not new. Um, transformers had been invented by that point, but like had not been, you know, broadly applied the way that they are today. So what, what is it that like this new wave of AI unlocked by things like by, by transformers and convolutional neural networks and so on? Like, what does that Enable above and beyond what you would have been able to do five years ago?

A Yeah, that's a good question. So the way I look at it is, so transformers are very similar to graph neural networks. Um, they, both of them are, so actually, let's, let's, let's take this back. So we used to use often convolutional neural networks, and the idea here is it learns a little function that's sort of local in an image. And then it sort of applies that same function everywhere, and then you stack up sequences of these layers, and that eventually lets you, like, one, the information on one side of an image communicate with the information on the other side of the image, because it's a hierarchy. A transformer architecture allows you to make a direct connection between the information on one side of the image and the other side of the image, the same way the graph neural network does. And I like to think about it like graph neural networks, because what it's like saying is, Well, in a graph, you have nodes, and you have edges or connections between the nodes, and a longer connection between nodes is for nodes that are farther away, and shorter connections are for nodes that are closer. So if you use the graph neural network analogy to describe the older convolutional networks, it's like the graphs are all small. They're all kind of, everything's kind of close. It's like nearby in an image. Graph neural networks allow you to choose how far away you want information to in…

AI assessment note: “A transformer architecture allows you to make a direct connection between the information”

Q There's a good comparison there. What do you think of as being, I guess, you just described something that is similar about what you can do in weather forecasting, thanks to a transformer architecture, to what you can do with large language models, which is what most people are going to be most familiar with in the, in the new wave of AI. What's different?

A So, that's a great question. I think what's, what's interesting is that the way that, so in, in language, the text is understood to be, is treated as a sequence. It's like, you know, token, token, token. We are also modeling sequences in weather, but we're not allowing our models to look too far back in time. So, because weather, weather is actually different from text in a fundamental way. Um, in fact, most physical processes are. They are what's called Markov, in that the most recent state of the system determines the subsequent state. So, like I said, in text, that's not the case. Uh, you know, right now, I'll just pause You didn't know what word I was going to say next, right? It kind of depends on the context, a bunch of words behind it or earlier. With weather forecasting in principle, if you know exactly what's happening right now, you can fully predict what's going to happen next. You don't need to look further back in the past. So we actually use transformers not to model the spatial, the interactions in weather over time, like the sequence of text, but in space. So in text, you actually don't have a sense of spatial structure, right? You just have one sequence of text. It's just word, word, word. When you read, you just see word, word, word. In weather, you have spatial structure. You have weather all over the Earth at the same time, and it's all, you know, especially…

AI assessment note: “weather is actually different from text in a fundamental way.”

Q gaps are. I mean, like, obviously we don't, we don't have the ability today to generate a perfect forecast three months into the future. Like it could always get better, but apart from just that element of it, are there any areas where you feel like actually there's like a real, it's really hard to do X? Is it like precipitation is a bugaboo or, you know, something else, right?

A Yeah. I mean, I would, I think there's sort of two ways to answer that. It's, you're always going to be limited by the quality of your data. So if you don't have good data about something, it's, it's going to, you know, you're just bad, you know, garbage in, garbage out sort of thing, right? So these models take an estimate of the state of the current weather and then predict what's going to happen. If your estimate isn't very good because your raw observations weren't very good, you're not gonna get a very good forecast. Um, so improving, just collecting more data, And using the data you have collected to form a better estimate of the current weather, that's definitely going to always improve things. So that's, it's sort of a known gap, right? Now we don't know exactly what the ceiling is. We don't know, like, if we've, you know, if we do this satellite or that station observation, how is it going to improve things? We might have an idea, but we don't always know, and sometimes we have to just test it out. Um, but the, um, yeah, the, the other thing I would say is that you, um, You have different features of weather which are harder or easier to predict. So an obvious one is temperature. So temperature is sort of very smoothly, like if you look at a map of the temperature across the earth, it's sort of, you know, it's not, you know, up a mountain, it's going to be colder, and …

AI assessment note: “different features of weather which are harder or easier to predict.”

Q finish by talking about what might come. Like, if you, if you draw a line forward a few years into the future, I know you pick, pick your time, three years, five years, 10 years, whatever it is, um, and, and you and everybody else who's working on AI weather forecasting succeed, um, Where might we be? Like, what might be possible in a few years that's not possible today?

A Yeah, I mean, I think that, I think there's a lot, you know, weather affects everything, and, um, it's, you know, it has, you know, different things. Energy is obviously a very, very sensitive to weather. Some things are, you know, only kind of, uh, you know, loosely affected by weather. Um, so one thing I would like to see, and I think it's very exciting, is a wider range of, um, use cases of weather. So, um, for example, like, we know that, uh, Even people make different choices about, you know, what to put in their refrigerator or like, you know, what clothes or whatever. These different choices are going to go on a trip, what they expect the weather to be. I think that you can start to make more subtle and informed, uh, kind of guidance and suggestions for people on the basis of what, uh, more accurate weather forecasts. And that's kind of like at the consumer level, but I also think that at the kind of, you know, industry level, there could be a huge opportunity. So for example, in energy, You know, we see there's, you know, if you have a wind farm or a solar farm, you're making forecasts about the weather, and then you're kind of using that to figure out, like, how, you know, if you're going to have energy to sell and how you're going to price it. But I have a feeling that there's a lot of headroom, a lot more to be gained in how we, you know, plan our, you know, how to o…

AI assessment note: “a wider range of, um, use cases of weather.”

Q as a subset of AI, a related entity. We've been doing ML already, so I guess the first question that I have is, um, as you think about leveraging AI now and into the future for weather forecasting, what version of AI are we talking about? Like, what version or versions are you actually, what is the, what are the actual Capabilities and or model structures that are interesting here?

A Yeah, that's a good question. So, yeah, these days, I mean, AI is a pretty catch-all term. Um, I, I find myself just using the word AI just to mean a lot of different things, because I think it's kind of easier, and usually people kind of know. Um, the difference, the way, but the way I would say it is, the difference between AI and, so machine learning is sort of the, like, statistical Inner core of AI. Um, it's trying to capture, it's taking data and trying to capture the patterns through a training process, and then, uh, you know, kind of use some inductive assumption like, oh, what we've seen in the past is going to be similar to what we see in the future. Um, Modern AI, I think, is a broader family of things. It sort of involves, like, you know, agents and your interactions with them, and a lot of, like, language models are often, you know, sort of associated with AI. Um, what we use in our weather forecasting models, and a lot of folks out in the community are using it as, as, you know, as this field is advancing and this AI-based weather forecasting is developing, we're still mostly using fairly traditional machine learning, supervised learning. So, supervised learning just means You take a data set that has a pair of, uh, examples, the, an input example and a target example, and you train a model to try to take input examples and accurately predict the target examples. …

AI assessment note: “we're still mostly using fairly traditional machine learning, supervised learning”

Q you school me a bit on something that I realize I don't know, which is how do we do weather forecasting? Like currently, and maybe a little bit of history, like what have, have there been major shifts technologically and how we forecast weather historically? So maybe walk me through the history such as it is Of how we forecast weather, and then, like, what do we actually do today?

A Yeah, so I have to admit, I'm actually a relative newcomer to the area of weather forecasting myself, so we had gotten involved in this, uh, several years ago, uh, and it was sort of built out of a research program that was trying to model complex simulations, including fluids, and the Earth's atmosphere is a fluid, and one of the big, uh, challenges that we were sort of interested in exploring was You know, modeling the atmospheric fluid, which is weather forecasting. So I, I should say that I sort of have gone through this, uh, uh, journey of learning about weather forecasting. So the stuff that I'll say, hopefully, hopefully it's accurate, um, but, uh, you know, forgive me if I make mistakes. So I think my understanding about the field is that really a lot of the, you know, historically weather forecasting was very important for agriculture and, uh, you know, sort of other use cases that were very important for kind of day-to-day life. Um, but I think it was about maybe A hundred, a 150 years ago that you had, um, agencies or bureaus that were starting to do, like, marine forecasting or, like, kind of more systematically collecting observations systematically and treating it as a science. Um, but then, probably about 50 years ago or so, you started to see the emergence of, like, large government public weather agencies. So, I think NOAA in the US was formed, uh, in the seven…

AI assessment note: “about maybe A hundred, a 150 years ago that you had, um, agencies”