CUDA, every mention
100 scenes · ← back to CUDA
tap a year for its mentions
every year anyone Ben Gilbert 51David Rosenthal 33Jensen Huang 16
Verbatim, from the transcripts: the passages where CUDA comes up
Google Part III: The AI Company. Google is amazingly well-positioned... will they win in AI? (Audio) · Acquired
- ▶ 50:33 David Rosenthal The Toronto team rewrites their neural network algorithms in CUDA, NVIDIA's programming language.
- ▶ 3:28:41 Ben Gilbert So if you're willing to not use CUDA and build on Google stack, they have an abundant amount of TPUs for you.
- ▶ 3:31:45 Ben Gilbert They're trying to build an ecosystem around their chips the way that CUDA does, and you're only gonna credibly be able to do that if your chips are accessible in anywhere that someone's running their existing workloads.
NVIDIA CEO Jensen Huang · Acquired
- ▶ 10:51 Ben Gilbert So is the lesson for founders out there, when you have conviction on something like the Revo-one-twenty-eight or, ah, CUDA, go bet the company on it.
- ▶ 12:08 Ben Gilbert Do you feel like that was the case with CUDA?
- ▶ 12:12 Jensen Huang In fact, before there was CUDA, there was a CG.
- ▶ 12:53 Jensen Huang And so there were a lot of characteristics about programmable shading that would suggest that CUDA has a great opportunity to succeed.
- ▶ 13:17 Ben Gilbert In building that platform.
- ▶ 14:51 Jensen Huang And the reason for that is because we were already working on computer vision at the time, and we were trying to get CUDA to be a good computer vision system.
- ▶ 14:58 Jensen Huang Or most of the algorithms that were created for computer vision aren't a good fit for CUDA, and so when we're sitting there trying to figure it out, all of a sudden AlexNet shows up.
- ▶ 18:38 Jensen Huang And we were fortunate that working with the world's universities and researchers was, was innate in our company because we were already working on CUDA and CUDA's early adopters were researchers because we democratize supercomputing.
- ▶ 18:52 Jensen Huang You know, CUDA is not just used, as you know, for AI.
- ▶ 18:54 Jensen Huang CUDA is used for almost all fields of science.
- ▶ 19:08 Jensen Huang And so the number of applications of CUDA in research was very high, and so when the time came and we realized that deep learning could be really interesting, it was natural for us to go back to the researchers and find every single AI…
- ▶ 30:41 Jensen Huang Somebody is, uh, build CUDA for Hopper.
- ▶ 30:44 Jensen Huang Somebody's job is build CUDNN for CUDA for Hopper.
- ▶ 36:26 Jensen Huang And our second data center product was remote graphics, putting our GPUs in, in the world's enterprise data centers, which then led us to our third product, which combined CUDA plus our GPU, which became a supercomputer, which then worked…
- ▶ 50:53 David Rosenthal They weren't, there was no CUDA.
- ▶ 51:08 Jensen Huang It's the OODA of CUDA.
- ▶ 51:10 David Rosenthal Could it compute unified device architecture?
- ▶ 53:14 Jensen Huang If you look now, when CUDA came along, there was OpenGL, there was DirectX, um, but there's, there's still another, uh, extension, if you will, and that extension is CUDA, and that CUDA extension allows a chip that
- ▶ 53:28 Jensen Huang Got paid for running DirectX and OpenGL to create an install base for CUDA.
- ▶ 53:36 David Rosenthal And is this why you are so militant, and I think from our research it really was you being militant that every NVIDIA chip will run CUDA?
- ▶ 54:01 Jensen Huang 250,000,300 million installed base of active CUDA GPUs being used in the world today, and they're all architecturally compatible.
- ▶ 54:30 David Rosenthal I mean, and I guess, ah, CUDA was a rebirth of UDA, but understanding this now, UDA going all the way back.
- ▶ 1:24:05 Jensen Huang Two, three billion dollars in market value for a while because of the decision we made and going into CUDA and all that work, and your belief system has to be really, really strong.
Nvidia Part III: The Dawn of the AI Era (2022-2023) (Audio) · Acquired
- ▶ 10:09 David Rosenthal They bought two GeForce GTX-Five-Eighty's, which were the top-of-the-line cards at the time, and they wrote their algorithm, their convolutional neural network, in CUDA,
- ▶ 1:13:26 Ben Gilbert And it all runs CUDA.
- ▶ 1:13:35 David Rosenthal Which means that whatever developers you already had who were working on AI or anything else, everything they were working on is just going to come right over and run within your brand new shiny AI supercomputer, because it all runs CUDA.
- ▶ 1:13:50 David Rosenthal More on CUDA in a minute.
- ▶ 1:16:42 Ben Gilbert It's got 18 and a half thousand CUDA cores.
- ▶ 1:16:54 Ben Gilbert This one H one-hundred, which they're calling A-GPU, has 18 and a half thousand cores that are capable of running CUDA software.
- ▶ 1:26:55 Ben Gilbert They own the developer relationship through CUDA.
- ▶ 1:37:23 Ben Gilbert We gotta talk about CUDA before we start analyzing anything else here.
- ▶ 1:37:37 Ben Gilbert And CUDA, as folks know, was the initiative started in 2006 by Jensen and Ian Buck and a bunch of other folks on the NVIDIA team to really make a bet on scientific computing, that people could use graphics cards for more than just…
- ▶ 1:38:11 Ben Gilbert So, CUDA has become the foundation that everything that we've talked about, all the AI applications, are written on top of today.
- ▶ 1:38:22 Ben Gilbert So, you know, you hear Jensen in these keynotes reference CUDA the platform, CUDA the language, and I spent some time trying to figure out, like, when I was watching developer sessions and, like, literally learning some CUDA programs, what…
- ▶ 1:38:40 Ben Gilbert So today, CUDA is, starting from the bottom and going up, a compiler, a runtime, a set of development tools like a debugger and a profiler.
- ▶ 1:39:03 Ben Gilbert And if you're a CUDA developer, your stuff works on everything.
- ▶ 1:39:50 Ben Gilbert If you look at the number of CUDA developers over time, it was released in 2006,
- ▶ 1:40:13 Ben Gilbert Twenty-twenty-two, they hit three million developers, and then just one year later, in May of twenty-twenty-three, CUDA has four million registered developers.
- ▶ 1:40:32 Ben Gilbert They don't think about it like, well, CUDA is our moat versus competitors.
- ▶ 1:42:46 Ben Gilbert And as you'd imagine, you need a whole new type of programming language and compiler and everything to deal with this new computing model, and that's CUDA, and it frickin' works.
- ▶ 1:46:52 Ben Gilbert I mean, if you go back pre-CUDA when they were a commoditized graphics card manufacturer, it was 24%.
- ▶ 1:48:21 David Rosenthal They're not just the best because of CUDA.
- ▶ 1:58:41 Ben Gilbert AMD doesn't have the developer ecosystem from CUDA.
- ▶ 2:00:50 Ben Gilbert This has CUDA written all over it.
- ▶ 2:01:04 Ben Gilbert You can justify whatever it is, 1600 people who actively on LinkedIn at NVIDIA today have the word CUDA in their job title.
- ▶ 2:02:25 Ben Gilbert Cuda competitor made by AMD for their hardware, but again, new, not a lot of adoption.
- ▶ 2:02:51 David Rosenthal iOS versus Android, because Nvidia has had first dozens and then hundreds and now thousands of engineers working on CUDA for 16 years.
- ▶ 2:03:52 David Rosenthal NVIDIA has thousands of engineers working on CUDA, 10 years ahead.
- ▶ 2:03:57 Ben Gilbert I sent you this graph, David, of my estimated number of employees working on CUDA per year since inception in 2006, and then if you look at the area under the curve and just take the integral, it's approximately 10,000 person years that…
- ▶ 2:11:44 Ben Gilbert I mean, remember, there are people building libraries on top of CUDA, and you can use the building blocks that other people built to build your code.
- ▶ 2:11:52 Ben Gilbert You can write amazing CUDA programs that just don't have that many lines of code because it's calling other pre-existing stuff.
- ▶ 2:12:05 Ben Gilbert But it looks genius in hindsight to make sure that every GPU that went out the door was fully CUDA capable, and today there are five hundred million CUDA capable GPUs for developers to target.
- ▶ 2:12:32 Ben Gilbert Include CUDA taking up a huge amount of space on this thing and make all these trade-offs in our hardware so that we can write, what, are people gonna use CUDA?
- ▶ 2:46:36 Ben Gilbert And even if you figured out how to do that, you'd need to build software that is as good or better than CUDA.
- ▶ 2:46:52 Ben Gilbert And even if you made all these investments and lined all of this up, you'd of course need to go and convince the developers to actually start using your thing instead of CUDA.
- ▶ 2:48:56 Ben Gilbert I mean, one, Ian Buck from NVIDIA, who leads the data center effort and is one of the original team members that invented CUDA way back when.
Holiday Special 2022 · Acquired
- ▶ 23:02 Ben Gilbert And maybe I'm forgetting the story in its entirety, but I think there was a bet the company moment around programmable shaders, and then a third one around the, like, seven-year investment before the market was there in AI and CUDA and…
Nvidia: The Machine Learning Company (2006-2022) · Acquired
- ▶ 19:16 Ben Gilbert Because while CUDA development began in 2006,
- ▶ 19:51 Ben Gilbert I searched LinkedIn for people who work at NVIDIA today and have the word CUDA in their title.
- ▶ 19:56 Ben Gilbert There are 1100 employees dedicated specifically to the CUDA platform.
- ▶ 21:33 David Rosenthal In 2006, 2007, 2008, they are pouring a lot of resources into building what will become CUDA that we'll get to in a second.
- ▶ 21:43 David Rosenthal Um, it already is CUDA at this point in time.
- ▶ 25:00 Ben Gilbert Because, and anything that CUDA empowers is not yet a revenue driver, and they've totally taken their eye off of gaming.
- ▶ 26:03 David Rosenthal So instead he goes and builds CUDA and continues to build CUDA.
- ▶ 26:10 David Rosenthal Like we get excited about a lot of stuff on acquired, but I think CUDA is like one of the greatest business stories of the last 10 years, 20 years, more.
- ▶ 27:05 David Rosenthal So what is CUDA?
- ▶ 27:08 David Rosenthal It is NVIDIA's Compute Unified Device Architecture.
- ▶ 27:49 Ben Gilbert Super high level application development, you know, really high abstraction layer development for hundreds of industries at this point to communicate down to CUDA, which communicates down to the GPU and everything else that they have done…
- ▶ 28:15 David Rosenthal Ben Thompson had this amazing interview with Jensen on strategy and, uh, Jensen in this interview, I think puts what CUDA is and how important it is.
- ▶ 28:29 David Rosenthal We've been advancing CUDA and the ecosystem for 15 years and counting.
- ▶ 29:47 Ben Gilbert Literally true because most programming languages up to this point and most computing platforms primarily contemplated serial execution of programs, and what CUDA did was it said, you know what, the way that our GPUs work and the way that…
- ▶ 30:18 Ben Gilbert So insanely, or dare I say, embarrassingly parallel, and CUDA is designed for parallel execution from the very beginning.
- ▶ 31:22 David Rosenthal NVIDIA to this day, now this may be changing, we'll talk about this at the end of the episode, has never charged a dollar for CUDA, but anyone can download it, learn it, use it, you know, blah, blah, blah, you know, all of this work stand…
- ▶ 32:12 Ben Gilbert So OpenCL is sort of the main competitor at this point, and they do actually let OpenCL applications run on their chips, but nothing in CUDA is available to run elsewhere.
- ▶ 35:29 David Rosenthal They're committed to continuing to invest in CUDA and making general purpose computing on GPU a thing.
- ▶ 47:49 David Rosenthal So AlexNet took these old ideas and it implemented them on GPUs, and to be very specific, it implemented them in CUDA on NVIDIA GPUs.
- ▶ 48:13 David Rosenthal This was the big bang moment for artificial intelligence, and NVIDIA and CUDA were right there.
- ▶ 48:37 Ben Gilbert In fact, they even thought about should we relaunch our GPUs as GP GPUs, general purpose graphics processing units, and of course they decided not to do that, but just built CUDA.
- ▶ 49:43 Ben Gilbert And this ends up being the very core of what becomes CUDNN, which is the library for deep neural networks that's actually baked into CUDA that makes it
- ▶ 57:06 David Rosenthal Well, I think people had just lost trust and interest, you know, after, like, there were so many years of, like, they were so early with CUDA and early, you know, they didn't even know that this, like, they didn't know AlexNet was gonna…
- ▶ 1:20:13 Ben Gilbert Over time they might do some stuff like that, but the thing that they were sort of like, which, which is believable, beating the drum on that the strategy was going to be, is right now our whole business strategy is that CUDA and…
- ▶ 1:21:04 David Rosenthal So they just did GTC at the end of March, the big, uh, developer, the big GPU developer conference that they do every year that they started in 2009 as part of building the whole CUDA ecosystem.
- ▶ 1:21:21 David Rosenthal Three million registered CUDA developers, 450 separate SDKs and models for CUDA.
- ▶ 1:25:33 Ben Gilbert Architecture, systems, data center, CUDA, CUDAX.
- ▶ 1:26:04 David Rosenthal They say that they're gonna start licensing a lot of the software that they make separately, licensing it separate from the hardware, like CUDA.
- ▶ 1:48:47 David Rosenthal This is 15 years of CUDA and the hardware underneath it and the libraries on top of it that Nvidia has built.
- ▶ 1:50:22 Ben Gilbert The whole CUDA investment.
- ▶ 1:51:06 David Rosenthal To the extent that you have lock in to developing on CUDA, which I think a lot of people really have lock in on CUDA, then that's major switching costs.
- ▶ 1:51:17 David Rosenthal Like, if you're gonna boot out NVIDIA, that means you're booting out CUDA.
- ▶ 1:51:21 Ben Gilbert Is CUDA a cornered resource?
- ▶ 1:53:31 Ben Gilbert I don't think it was for CUDA.
- ▶ 1:53:46 Ben Gilbert That Nvidia gets by owning not only the driver stack, but, you know, all of CUDA and how tightly coupled their hardware and software is.
- ▶ 1:54:03 David Rosenthal CUDA and NVIDIA and general purpose computing on GPUs is a platform, so whatever, you know, all of the stew of powers that go into making that, that go into making Apple, Microsoft, you know, and the like,
- ▶ 2:08:00 David Rosenthal It's like, hey, we're building this CUDA thing.
Nvidia: The GPU Company (1993-2006) · Acquired
- ▶ 1:15:43 Ben Gilbert Nvidia's approach was to what graphics cards had been before, and building programmable shaders, and creating CG, which was a little bit of an early strategy and something they would later do with CUDA, but really understanding that like,…
- ▶ 1:37:23 David Rosenthal I mean, they made CG in collaboration with Microsoft, and CG works on NVIDIA products, but it's not like CUDA today to flash forward to next time.
- ▶ 1:56:16 Ben Gilbert Then the three D chess version is, and this kind of foreshadows the next episode because Nvidia had to learn these hard lessons and had to develop, like was forced to develop these really crazy competencies, like eventually developing CUDA.