The Wisdom Wall

“you cannot be, like, checking out on, like, Friday, Saturday, Sunday, and, like, work at, like, nine to five if you want to, like, Make progress, or like, some people are just so good at detaching, like, ok, like, like, you know, like, eight pm, I'm not going to, my job can die, and then the chips can stay idle for,…”

“It's not architecture itself. That's, that there's a problem that we, that is more of like the learning paradigm itself rather than the architecture itself. I think the architecture is just basically like the interface between the learning algorithm and the tokens. I think it's more about the learning algorithm itself.”

“the bitter lesson gets used too much in, like, too conveniently used around, but actually there's also a little bit of a, not a bit, there's also a sweet lesson where it's like, ideas matter.”

“Usually when, like, a provider, like, provisions new notes, or they would, like, uh, give us... Yeah, it's usually, like, bad, like, like, dog shit, like, like, at the start. Uh, and then, uh, it gets, like, better as you go through the process of, like, returning notes, like, and, and, you know, like, uh, like,…”

“So I think on policyness is basically this idea of like model training on its own outputs and letting the model like generate its own trajectories and then letting some reward verify it and then the model train its own outputs. I think this is more generalizable in general.”

“these things are not like, like going to replace one person as it is, but more like a passive aura that buffs everybody.”

“I think that it's true that sometimes a lot of progress on the whole is just a series of small incremental changes that, yeah, that push. I think that's accurate. That's true. There's also, it also feels that there's also a lot of like small, like seemingly minor for the lack of better word, like that push AI to the…”

“So if you are, you come to a point where you are Very data bound, but not compute bound at all. You just find algorithms that spend a lot of compute on every token.”

“ML. ML can be learned easily. Our knowledge can be learned easily.”

“if you do it like a, like, compute startup or anything, the biggest green flag would be to, to share the cause of node failures with, ah, with, ah, your, your customers, right”

“the only big benefit of encoder decoders [4425] Yi Tay: Is that it has this thing called, like, I mean, what I like to call intrinsic sparsity. [4430] Podcast Host: Ok. [4431] Yi Tay: So basically, an encoder decoder with, like, n baramps is, like, uh, like, basically, if it's, like, it has the cost of, like, a n over…”

“serious LMS that create their own evals, and they, a good eval set is one that you don't release. A good eval set is the one that you, like, ok, you release some of it, but, like, it's like, you don't, like, you know, let it be contaminated by the community.”

“To me, Lama Tree is like... Meta has an org that is hypothetically very similar to Gemini or something but they just decide to release the weights It's open weights It's open weights and everything”

“when people realize that, like, this, like, turning on the GPT-IV tab and running some DPO is not going to give them the reward signal that they want anymore, right? Then all these variants gone, right? You know, there was this era where there's, wow, there's so many of these, like, I can't remember, I lost track of…”

“You actually don't have to follow anything. If the paper is important enough, the Twitter algorithm will give it to you.”

“I don't think there's actually much, like, the government can do to, like, influence, like, this kind of thing is, like, a natural, like, organic, natural thing, right? Uh, the worst thing to do is probably, like, to create, like, like, create a lot of artificial things that, like, uh, Exchange programs?”

“So I think the actual, like, technical definition of reasoning is making models better with thinking and post-training. Ok? Yeah. So basically, like, RL-ing the model to think better.”

“There was this whole big era, which I was also involved in this era, where people try to like undermine the attention as much as possible. Like they try to remove it, simplify it, make it efficient, like this whole Like, like, efficient attention era. At the end of the day, the outcome was always like, oh, we removed…”

“if somebody comes up with something and then does something that you feel that is very tasteful and it aligns with what Like researchers in the labs, like, like one, and they come up with that independently, you know that the function that is good, right? Like if you just go and tell somebody to do this, like you, you…”

“I don't think there's any specific thing. In fact, I will try to be as, like, like, agnostic, like, I don't, like, I don't really say, like, okay, you need to learn JAX, you need to learn this, by the time you finish learning, there's a new framework out, anyway. So, so it's more of, like, staying, like, constantly,…”

“a lot of architecture changes, right, the moment they are, like, tedious to implement, like, nobody, like, SuiGuru is a simple thing, right? [4306] Yi Tay: Just split it and then get it. [4307] Yi Tay: It's a very simple thing to implement. [4309] Yi Tay: Maybe that's why it's caught on, because it has, like, a, a, a…”

“I think in the ideal case, you do MOE from scratch.”

“I usually start writing the thing while working on that thing itself. Like, so, even, like, let's say, like, like, if you want to launch something, like, then the end goal is, like, a blog post, or shipping something, everything, right? I like, or not really a launch, or, like, like, that's papers, or I always like to…”

“keyboard is more optimal than moving your head because If you can switch your screen fast enough, it's faster than your head, like, moving to different screens, and stuff like that.”

“If there's one bad note, it kills the entire job, right? So, like, the fact of, like, the game became, like, just eliminating bad notes from the, from the thing, right?”