PyTorch

“The correctness for both forwards and backwards passes is there, but on Nvidia, it's about five X slower than PyTorch right now.”

“Replicating the PyTorch API. Is something I can do with a couple, you know, like an engineer month or two.”

“The metaphor here is that you don't need to be digging down into like this sort of PyTorch level if you don't want to in the same way as a software engineer in the nineties. You don't need to be like understanding how network stacks work to be able to build a …”

“It's used in Mars rover simulations, to drug discovery, to Tesla cars, and there's a huge diversity of, like, applications in which it is used in.”

“If they end up expanding onto the server side, and they'll probably build something like PyTorch as well, right? Like, eventually, that'll where it will land. And I think there, they will kind of fail on the, like, lack of differentiation. Like, it wouldn't be…”

“Mojo as a fundamental frontend would be replacing PyTorch, not, like, augmenting PyTorch. So, in that sense, I don't see a synergy in more deeply, like, integrating Mojo.”

“I think the scariest was when I went to visit CERN at some point, and they said they were using it, PyTorch, and they were using GANs at the same time for, like, particle physics research, and I was scared more about the fact that they were using GANs than the…”

“There's no, like, I don't think, like, unless we have, like, great breakthroughs, like, George's vision is achievable, like, or, like, he should be thinking about a narrower problem, such as, I'm only gonna make this for, like, work for self-driving car con ne…”

“PyTorch is like a 190,000 lines of code or something at this point.”

“Easy context was the first PyTorch implementation that applied it with native libraries that worked pretty well. And then we adapted it ourselves in order to configure it for our cluster network topology.”

“A lot of architecture changes, right, the moment they are, like, tedious to implement, like, nobody, like, SuiGuru is a simple thing, right? [4306] Yi Tay: Just split it and then get it. [4307] Yi Tay: It's a very simple thing to implement. [4309] Yi Tay: Mayb…”

“So, PyTorch is really, really nice, and this is just some of the things that PyTorch offers. So, without PyTorch, we're kind of naked in the world, right?”

“The use of Python and PyTorch and everything else is just a crutch, because we humans are finite. We have finite knowledge, intelligence, and attention.”

“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”

“PyTorch actually started as the framework for researchers. Don't care about production at all.”

“If we decided to change one thing in PyTorch, like one line of PyTorch code is like a week of CUDA code at least.”

“So basically October, 20, 22, just like I hadn't written a line of PyTorch before that. So it's completely new.”

“The problem with PyTorch is it's high levels. Don't have that much fine-grained control over, like specific memory stuff, so kind of have to leave, like, the memory freeing to Python and PyTorch, which is, can be annoying sometimes.”

“And so things like PyTorch came on the scene and I think PyTorch gets all credit for democratizing model training, right? It's taught to pretty much every computer science student that graduates. That's a huge deal, but nobody democratized inference. Inference…”

“And AMD had their own separate you know, non-mainline PyTorch distribution for years. That was always six to nine months behind any new PyTorch releases.”

“Most of that is an attention and MOPs, right? So if you have good kernels for attention, MOPs and norms and so on, then it doesn't much matter what the front end to, you know, send tensors to and from those kernels is”

“Agents are a distinct enough set of technology that it merits its own community. [5135] Swyx (Shawn Wang): Separate from data and AI. [5136] Jim Zemlin: Yeah, because like a PyTorch dev isn't really doing a ton of stuff in agent land, right?”

“So what it really enables is that you can now write neural networks essentially in Lean. So instead of writing in, like, PyTorch, it's like a PyTorch-like abstraction, but you can, like, kind of, you know, write it in Lean, and so it can be fully formalized in…”