The Wisdom Wall

“On the other hand, if you think about using transformer architectures that have worked so well for language, that just wouldn't be able to support a five trillion context length. No matter all the compute in the world is thrown at it. So that kind of quadratic complexity is infeasible and also unnecessary because the…”

“And the difference there is compared to language where self-improvement needs something like human feedback or other reward signals that are very sparse. They just tell you yes or no, thumbs up or down. We have dense feedback because the physics laws, there's so multiple of them, and you can decide again a curriculum…”

“And so this reliance on just purely data-driven AI is not going to be enough. And that's where, you know, adding the loss of physics is really critical.”

“So there's implicitly a lot of common features, even across physics that are gone by different equations. So that's how you see across different domains, across different mathematical models. There's a lot of Shared features that these neural models can pick up.”

“optimization ends up being usually very difficult, especially for problems that are time-dependent, meaning it's not just stationary, you also have time, and the time component in many cases could be turbulent, like in the case of fluid dynamics, you know, you kind of, like, if you Run it long enough. It can become…”

“neural operators enable because they model inputs and outputs as continuous functions that can be infinitely resolved, that can have infinite discretization. And now we can have, uh, You know, at inference time, you can give it now inputs and ask for outputs at any resolution. So you're not just limited to the…”

“if you now give it the model additional information in terms of, let's say, a physical loss, so you could give it partial differential equation constraints, conservation laws, and you can now enforce them at a finer resolution than the data you have, then there's more guidance in a way. So that way it can now come up…”

“if we were to use transformers and we require a very high resolution, it would become untenable because of the quadratic complexity and all to all connections. On the other hand, if you did that with Fourier transforms, we have like quasi linear, uh, complexity and still we have global connections in a way we can model…”

“a lot of data, purely data driven approaches in a way seeing saturation, right? So now we want to ask, okay, either make them more hardware efficient, right? There's a lot of now room to kind of say, can we now, you know, make them much more energy efficient or hardware efficient? So that's one aspect. But the other is…”

“neural operators have this flexibility because they can allow you to mix across different scales within the model rather than be prescribed externally like a lot of other Hybrid machine learning for physics do, and that, you know, allows us to be a lot more data efficient and much better at learning.”

“Yes, you can do a lot of hypothesis generation. You can have ideas, but ideas are not enough, right? So you can have a lot of ideas. The bottleneck is going, testing, and verifying that they work in the real world.”

“our idea of neural operators came as a way to overcome this, right? So saying, you know, we can't rely just on physics constraints alone to come up with answers. We have lots of data available. Um, you know, I'll talk about the weather example where we even collect data, right? So we don't just solve equations and Have…”

“So that's one way of thinking because, you know, first of all, we are lifting the signal to more dimensions, even if the signal is two or three dimensions, uh, we are now lifting it to much higher dimension. So in that space, the idea is it's easier to learn and we're doing it as a nonlinear lifting, right? So there's…”

“But I think this is where more broadly the lesson is the physical world may be more forgiving because, you know, where there are extreme events like hurricanes that have very specific physical signature. Right. So it's like extreme, but in a very specific way. So maybe you don't need as many samples because the…”

“making it a hard constraint is not tractable, whereas adding it as a loss function. And of course, there's still the balancing of that loss with the data we have.”

“So the idea of like a latent space to handle all kinds of different geometries and, and be able to capture the physics there in the latent space well, means we can now have a model that generalizes across a lot of different geometries.”

“And now there's coupling, like because of heat, there's also stretching or kind of the joint phenomena. You could like now hope to fine tune with much fewer samples because it kind of individually knows this phenomena. Then combining them together, maybe it can't do it from scratch because that's still too much to ask.…”

“And humans are usually not good at this, right? We are not good at like looking at highly nonlinear phenomena and say, oh, somehow maybe this combination of all these gates coming together helps pull the electrons together in a quantum gate. And so our collaborators were struggling to do that manually. And with AI, we…”