PyTorch

“So what it really enables is that you can now write neural networks essentially in Lean. So instead of writing in, like, PyTorch, it's like a PyTorch-like abstraction, but you can, like, kind of, you know, write it in Lean, and so it can be fully formalized in…”

“You build a LLM completely from scratch. You use PyTorch, but you build one from scratch that, you know, can be a chatbot. You train it on data. You RL it to solve math problems with tool calls. You do all of this. And this is a undergrad level course.”

“The JAX team at Google got into a giant fight with the PyTorch team on Meta on X, and the heads of each company's respective AI division had to make a public truce and instruct their troops to stop fighting.”

“Agents are a distinct enough set of technology that it merits its own community. [5135] Swyx (Shawn Wang): Separate from data and AI. [5136] Jim Zemlin: Yeah, because like a PyTorch dev isn't really doing a ton of stuff in agent land, right?”

“Most of that is an attention and MOPs, right? So if you have good kernels for attention, MOPs and norms and so on, then it doesn't much matter what the front end to, you know, send tensors to and from those kernels is”

“In inference, the truth is, nobody cares about CUDA. Nobody even cares about PyTorch. All right. What they want is an API.”

“We don't want to outsource this to some package because A, we're about to go to a bigger scale, like PyTorch, for instance, they had a package for doing this. But we were going to go to a bigger scale than Facebook had been to. And you don't want to have a dep…”

“The PyTorch profiler was, like, pretty good, actually, throughout for a single GPU. You want to, like, profile a GPU, the PyTorch profile would work. But if you wanted to profile a job on 100,000 of GPUs, that, like, hadn't really been done much, and then that…”

“And AMD had their own separate you know, non-mainline PyTorch distribution for years. That was always six to nine months behind any new PyTorch releases.”

“And so things like PyTorch came on the scene and I think PyTorch gets all credit for democratizing model training, right? It's taught to pretty much every computer science student that graduates. That's a huge deal, but nobody democratized inference. Inference…”

“It took us five years. Took us five years to get the stage supporting almost all internal needs using deep learning and mass and massive scale.”

“OpenAI switched to use PyTorch fully.”

“I think one of the biggest success we saw from the PyTorch experience is simplicity scales.”

“We have hundreds of engineers building PyTorch and infrastructure around PyTorch, but at the same time, I believe PyTorch within Meta probably has thousands of users.”

“So basically October, 20, 22, just like I hadn't written a line of PyTorch before that. So it's completely new.”

“The problem with PyTorch is it's high levels. Don't have that much fine-grained control over, like specific memory stuff, so kind of have to leave, like, the memory freeing to Python and PyTorch, which is, can be annoying sometimes.”

“If we decided to change one thing in PyTorch, like one line of PyTorch code is like a week of CUDA code at least.”

“PyTorch actually started as the framework for researchers. Don't care about production at all.”

“So we revolutionized deep learning because of our domain specific library called QDNN. Without QDNN, nobody talks about QDNN because it's one layer underneath PyTorch and, you know, and TensorFlow and back in the old days, CAFE and Theano and now Triton, and t…”

“But the one monolithic thing was actually just like a technical bot. Like it was from the fact that like we had PyTorch and then they like, you know, or like at Karis or any of the other ones. And they're like, Well, if you want, you can train on multiple GPUs…”

“So, PyTorch is really, really nice, and this is just some of the things that PyTorch offers. So, without PyTorch, we're kind of naked in the world, right?”

“At the time of that post, we were using, in LL and that's in 30% less memory, and we were 20% faster in training, just the truth.”

“The use of Python and PyTorch and everything else is just a crutch, because we humans are finite. We have finite knowledge, intelligence, and attention.”

“A lot of architecture changes, right, the moment they are, like, tedious to implement, like, nobody, like, SuiGuru is a simple thing, right? [4306] Yi Tay: Just split it and then get it. [4307] Yi Tay: It's a very simple thing to implement. [4309] Yi Tay: Mayb…”

“Easy context was the first PyTorch implementation that applied it with native libraries that worked pretty well. And then we adapted it ourselves in order to configure it for our cluster network topology.”

“PyTorch is like a 190,000 lines of code or something at this point.”

“There's no, like, I don't think, like, unless we have, like, great breakthroughs, like, George's vision is achievable, like, or, like, he should be thinking about a narrower problem, such as, I'm only gonna make this for, like, work for self-driving car con ne…”

“Mojo as a fundamental frontend would be replacing PyTorch, not, like, augmenting PyTorch. So, in that sense, I don't see a synergy in more deeply, like, integrating Mojo.”

“If they end up expanding onto the server side, and they'll probably build something like PyTorch as well, right? Like, eventually, that'll where it will land. And I think there, they will kind of fail on the, like, lack of differentiation. Like, it wouldn't be…”

“It's used in Mars rover simulations, to drug discovery, to Tesla cars, and there's a huge diversity of, like, applications in which it is used in.”

“I think the scariest was when I went to visit CERN at some point, and they said they were using it, PyTorch, and they were using GANs at the same time for, like, particle physics research, and I was scared more about the fact that they were using GANs than the…”

“We're, One of two qualified offerings on on PyTorch, and so all of that testing is being done you know, routinely with the regression testing that's run literally every night on any software release.”

“The metaphor here is that you don't need to be digging down into like this sort of PyTorch level if you don't want to in the same way as a software engineer in the nineties. You don't need to be like understanding how network stacks work to be able to build a …”

“Obviously with investment in Langtrain and we have a few others at Benchmark Cerebrus, which is you know, an AI chip for training and inference. We backed a team that worked on and was the leads of PyTorch and has got a new company and some others.”

“I don't think they really, I think people tell this, the story of sort of the silver bullet. Of like you know, the eager execution model. But I think the reality is they just built a product with so much more empathy.”

“So PyTorch really without PyTorch, none of this stuff would be happening right now. And so it's really like fundamental for all of the AI breakthroughs.”

“The correctness for both forwards and backwards passes is there, but on Nvidia, it's about five X slower than PyTorch right now.”

“Replicating the PyTorch API. Is something I can do with a couple, you know, like an engineer month or two.”

“ChatGPT was developed on PyTorch. Okay. All OpenAI runs on PyTorch. The entire world, in fact, runs on PyTorch, except Google, because they have their own thing, right?”

“We landed on using a framework that enables tree-based models. And we picked light GBM for that after trying a few different packages and also deep learning. And for that, we then used PyTorch. And so we started with those two core libraries.”

“We're definitely going all in as PyTorch, you know, end to end. So I think initially when we launched the Onyx strategy, it was more like a multi-framework world. And we had actually two framework internally between PyTorch and Cafe Two, but we're still suppor…”

“They have what I've, I would describe as a sandwich problem, which is that you end up sandwiching a lot of system code with user code, and then, like, adding more and more layers of that, and as you've sandwiched, like, you know, seven or eight layers of that,…”