Non-linearities enable neural operators to expressively capture limited frequency modes
Anima Anandkumar · 🔬 Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar, Caltech · Aug 26, 2026 · at 26:08
Caltech professor Anima Anandkumar explains how Fourier Neural Operators bypass classical signal processing frequency limits using non-linear latent spaces.
0:00 / 0:39exact quote · 39.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“So that's one way of thinking because, you know, first of all, we are lifting the signal to more dimensions, even if the signal is two or three dimensions we are now lifting it to much higher dimension. So in that space, the idea is it's easier to learn and we're doing it as a nonlinear lifting, right? So there's already a latent space there. And then we are doing further nonlinear transformations in between our Fourier transforms. So that means we are saying yes, you know, maybe with these limited number of frequency modes, it's not expressive enough, but when I add nonlinearities, I can, you know, I can kind of nice, more nicely capture that.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Anima Anandkumar
Transformers will never scale to high-resolution 4D physics simulations
“So forget ever having a transformer for anything of this scale. All of the world's compute will not be enough. And first of all, they all have to be co-located to be able to ever do this. So that's why we need other architectures.”
Neural operators are the only AI architecture that works for climate emulation
“This is where the Allen AI Institute has now built climate models based on our neural operator architecture. And that's the only one that works As an AI emulator, right? None of the other architectures work for climate because climate requires us to assume the…”
AI models predict fusion reactor plasma disruption one million times faster
“You know, I talk about plasma and fusion reactor. You know, we barely have a few thousand samples, but we are able to accurately predict events like disruption very well. And we are able to do that a million times faster than what traditional simulations were …”
Anandkumar: Multi-physics models outperform single-physics models of equivalent parameter size
“And in fact, I was going to add that it turns out that having the model of the same size with multiple areas of physics does better than giving all of those parameters to each single physics. So if you had separate models and made them big enough as the origin…”
Anandkumar: Standard Transformers cannot scale to 5-trillion context lengths for physics
“On the other hand, if you think about using transformer architectures that have worked so well for language, that just wouldn't be able to support a five trillion context length. No matter all the compute in the world is thrown at it. So that kind of quadratic…”
Anandkumar: Dense physics feedback enables better AI self-improvement than sparse LLMs
“And the difference there is compared to language where self-improvement needs something like human feedback or other reward signals that are very sparse. They just tell you yes or no, thumbs up or down. We have dense feedback because the physics laws, there's …”