Answered raw tape
D 5 · C 5 · P 4 · Cm 4 4.60
Q Um, that, that is actually a really nice segue. If you don't mind talking about liquid a little bit. Uh, you didn't mention liquid at all here. Uh, maybe introduce liquid to, to a general audience. Like what, uh, you know, what, how are you making an innovation on function approximation? Like what?
A Uh, the core idea of liquid neural networks is that the perceptron is not optimally expressed. In some sense, you can imagine that, uh, neural networks are a series of dams. They're pooling water at even intervals. And, ah, this is how we compute. But imagine that instead of having this static architecture that is only, ah, using the individual compute units in a very specific way, you, ah, have a continuous geography, and the water is flowing every which way. Like a river is parting based on the land that it's flowing on, and you can merge and pull, and even flow backwards. Ah, how can you get closer to this? And the idea is that you can represent this geometry using differential equations. And so by, uh, using differential equations where you change the parameters, you can get your functional approximator to follow the shape of the problem in a more fluid, liquid way. And a number of papers, uh, on this technology, and, um, it's a combination of multiple techniques. Um, I think it's something that ultimately is becoming more and more important and ubiquitous as, um, a number of people are working on similar topics.
AI assessment note: “the core idea of liquid neural networks is that the perceptron is not optimally expressed.”
Answered raw tape
D 5 · C 4 · P 4 · Cm 4 4.30
Q Okay, maybe I'll ask one more thing on this, which is, um, what are the interesting dimensions that we care about, right? Like, obviously you care about sort of open and maybe less child-proof models. Um, are we, are we, like, what dimensions are most interesting to us? Like, perfect retrieval, uh, infinite context, uh, multi-modality, multi-lingual, linguality, like, what dimensions matter?
A What I'm interested in is, uh, models that are small and powerful, but are not distorted. And, ah, at the moment we are training models by putting, um, basically the entire internet and the sum of human knowledge into them, and then we try to mitigate them by taking some of this knowledge away, but if we would make the models smaller, at the moment they would be much worse at inference and generalization. Yes. And, ah, what I wonder is, and it's something that we have not translated yet into, ah, practical, ah, applications, it's something It is still all research. It's very much up in the air. I think you're not the only ones thinking about this. Uh, is it possible to make models that represent knowledge more efficiently? And then basically epistemology, but it's the smallest model that you can build that is able to read a book and understand what's there and express this. Yeah. And, uh, also, maybe we need general knowledge representation rather than having, um, token representation that is relatively vague and that we currently mechanically reverse engineer to figure out if mechanistic interpellability, what kind of circuits are evolving in these models. Can we come from the other side and develop a library of such circuits that we can use to describe knowledge efficiently and translate it between models? You see, the difference between, uh, model and knowledge is that the, …
AI assessment note: “What I'm interested in is, uh, models that are small and powerful”
