NVIDIA

“We do think they're taking market share, especially in digital markets, like where they would be selling to an Intel or an Apple or an Nvidia. We think they're the share gainer, but both companies, it's like an extremely high quality duopoly.”

“NVIDIA has got close to 70% gross margins and like, you know, low forties operating margins. I did really with very little cyclicality because TSMC offloaded all that cyclicality. Right. So it is just like one of the best business models besides I would say en…”

“Literally 89 other companies get funded within a couple months to go do the same thing.”

“They have to lay off 70% of the company, which they do. They go down to about 35 people.”

“Imagine if you started a business in the early nineties doing a thing that seemed like a small market at the time, but you did the thing, and then it turns out that that gave you line of sight to something That the same technology was uniquely able to do That …”

“They sell their GPUs at a 66% gross margin. Hardware business with a 66% gross margin. Back in 2004, that gross margin was only 29% that they were able to command as a premium on their cards.”

“Throughout the company's history, they basically have these, like, six to 10 year epochs, and during those, they have, like, A meteoric rise when they do something contrarian that's off the rest of the industry, and then it starts to taper, and they need to fi…”

“When all the VCs funded these 90, you know, Silicon Valley startups to go make graphics cards, three D graphics cards, the only two surviving ones were Nvidia, which went through this hellish journey, and then these Canadian guys that were, like, totally out o…”

“And as hyped as Nvidia has been, you know, over the last five years, obviously with the stock run up and everyone's excitement around the company, I think Jensen is still an underrated CEO, even rated. Where the Nvidia bulls have put him.”

“Which makes them the fastest semiconductor ever to reach a billion in revenue and gets them added to the S&P 500.”

“They sell one million units of the Riva one 28 within four months.”

“And it's because this card that they ship, the first GeForce in 1999, it's the GeForce two 56. It's so powerful. It has five X better graphics performance than like anything else on the market.”

“Two million dollar total round. I don't know who invested what. I assume a million each, but two million dollar total round at a six million dollar post money valuation.”

“They go public in beginning in 1999 at a six hundred million dollar market cap. So a hundred X return from the six million dollar post money valuation on the Sequoia and Sutter Hill round”

“They had to spend a million dollars just to get the emulator, you know, hardware and software to do this.”

“NVIDIA is now on a cycle starting in 1997, 1998. Where they are doubling the performance that they are delivering at a given price point to the market every six months.”

“I think a lot of the GPs at Sequoia and certainly Mark Stevens, who was one of my professors at GSB. Who was on the board for Sequoia is still on the board have held their shares personally for like to this day. Like that's one of the best venture investment r…”

“AMD tried to buy Nvidia to make that their graphics line, and it was possible because it's not like the stock was blowing up at this point in time, and it had this sort of few years of reasonable stagnation before we get into late 2006, 2007, and certainly peo…”

“The next year, the two companies sign a huge multi-year deal for TSMC to become the primary foundry for NVIDIA and still are today.”

“The co-founder and CEO Jensen Huang really has bet the company, like the whole company, three separate times, nearly going bankrupt each time.”

“NVIDIA is the first. They are the first dedicated graphics card company.”

“It is the eighth largest company in the world by market cap.”

“They did raise some more money in between that initial two million and going public. I think they raised twenty million in total, but like there wasn't a lot of window and I think it was Sequoia and Sutter Hill that put that capital in for the rest of that twe…”

“Five hundred million dollar a year deal for Nvidia to supply the graphics for the Xbox with a two hundred million dollar advance.”

“When Nvidia began in 1993, it made computer graphics chips in a brutally competitive and low-margin market. There were 90 undifferentiated competitors all doing basically the same thing at the same time, and yet today, they have an 83% market share of standalo…”

“They land a big deal with Sega to power their arcade consoles and their next generation home console to be the three D graphics engine and would ultimately become the Sega Saturn.”

“Nvidia only has a billion dollars of capex per year.”

“Nvidia has literally three times the price to sales ratio of Apple, or price to revenue as Apple, and nearly two x Microsoft.”

“They go from sub a billion dollar market cap company after the tech crash up to five to six billion dollars kind of by 2004, 2005. Stock keeps going on a tear. By mid 2007, the stock reaches just under twenty billion dollar market cap.”

“So literally NVIDIA stock in 20, like recent, like this is now known, 2012, 1314, 15. It doesn't trade above like five bucks a share. And NVIDIA today, as we record this, is I think about two 20 a share. The high in the past year has been well over 300.”

“And so, in 2013, Brian Catanzaro, who's a research scientist at NVIDIA, published a paper with some other researchers at Stanford, which included Andrew Ng, where they were able to take this unsupervised learning approach that had been done inside the Google B…”

“Because I think now that we have the compute, the ability to gather the data, and the ability to actually, you know, run these simulations in a way that has a efficient way of running it and a good user interface to understand the data, people are going to sto…”

“Because while CUDA development began in 2006, That was not a useful usable platform for six plus years at NVIDIA.”

“So then insanely over the last two years, it three X the data center, the segment three X it is now doing over 10 and a half billion a year in revenue. And it's basically the same size as the gaming segment.”

“All the other graphics cards companies at the time and most peripheral companies, they let the further downstream partners rate the drivers for what they were doing. NVIDIA was the first one that said, no, no, we want to control this. We want to make sure cons…”

“NVIDIA is a platform, in my mind, no doubt about it. CUDA and NVIDIA and general purpose computing on GPUs is a platform, so whatever, you know, all of the stew of powers that go into making that, that go into making Apple, Microsoft, you know, and the like, G…”

“You do any of this work, you cannot deploy it on anything but NVIDIA chips.”

“The whole CUDA investment. Not at first, but definitely now is predicated on being able to amortize that a thousand plus employees spend over the base of the three million developers and all the people who are buying the hardware to use what those developers c…”

“The A plus is definitely that it does exist in the physical world, and they are the dominant provider of everything you need to be able to accomplish that. And if the real world stuff, you know, these little robots that run around factory floors, and the auton…”

“This is Google and Facebook and a whole list of other companies on the consumer side, and they're all dependent on NVIDIA.”

“NVIDIA to this day, now this may be changing, we'll talk about this at the end of the episode, has never charged a dollar for CUDA”

“So, you know, we said the high was around a twenty billion dollar market cap. It drops 80%. Eight zero.”

“They now have a 66% gross margin, so that illustrates to me how seriously differentiated they are and how much of a moat they have versus competitors in order to price with that kind of margin.”

“It powered the original Tesla Model S touchscreen. So, like, before any of the autopilot autonomous driving stuff, they were the processor powering just the infotainment, the touchscreen infotainment in the Model S.”

“The Tegra platform still to this day is the main processor of the Nintendo Switch.”

“It's not until 2016 that Nvidia gets back to the twenty billion dollar market cap peak that they were in 2007 when they were just a gaming company.”

“So I suspect that shifts more over time. I can't imagine a company that wants as much control as NVIDIA does loves the ad and board partner thing, but they've built a business on it, and so they're not really willing to cannibalize and alienate. But I bet if t…”

“We cannot overstate the importance of this moment, not just for NVIDIA, but for, like, computer science, for technology, for business, for the world, for us staring at the screens of our phones all day every day. This was the big bang moment for artificial int…”

“In 2018 they actually do change the terms of the user agreements of their consumer cards, of GeForce cards, that you cannot put them in data centers anymore.”

“Marc Andreessen gave an interview. He said, we've been investing in a lot of companies applying deep learning to many areas, and every single one effectively comes in building on NVIDIA's platforms. It's like when people were all building on Windows in the nin…”

“This is 15 years of CUDA and the hardware underneath it and the libraries on top of it that Nvidia has built. To go recreate that and surpass it on your own is such an enormous, enormous bite to bite.”

“Today there's over 10,000 cores on their most recent consumer graphics card.”

“They're growing at 60% a year. This is a thirty-year-old company that grew 60% in revenue last year.”

“Another one that I wanted to point out, it's a freaking hardware company, I know we didn't, they're not a hardware company, but they're a hardware company, with 37% operating margins, so this is even better than Apple”

“And then when the crypto winter hit in 2018 and the end of the ICO craze and all that, the mining rig demand It fell off and this had become so big for Nvidia that their revenue actually declined.”

“Oftentimes when someone has hardware that is differentiated by software and services, and then they decide to start selling those software and services a la carte, it's a strategy conflict. It's your classic vertical versus horizontal problem, unless you are g…”

“Jensen did not plan Alex net or see it coming cause nobody saw Alex net coming.”

“I searched LinkedIn for people who work at NVIDIA today and have the word CUDA in their title. There are 1100 employees dedicated specifically to the CUDA platform.”

“Those were big bets relative to the company's size at the time, but this bet is like an iPhone-sized bet.”

“I think Hopper is going to be the world's first four nanometer process chip using TSMC's new four nanometer.”

“What they did was they artificially crippled the card to make it worse at crypto mining.”

“It was 70 or 80 separate companies making graphics cards all got funded”

“When you look at, like, what NVIDIA's been doing with GPUs, it is totally fair to say that this 10 X improvement in computing every five, six, seven years, that is absolutely still happening”

“The business history of Qualcomm, just like Bill Gurley said on the blurb of this book, It is one of the most brilliant strategic executions of entering a market period. You know, like, writ large ever. Like, this is on par with NVIDIA, if not, honestly, more …”

“Qualcomm today is the largest fabulous semiconductor company in the world... Bigger than NVIDIA.”

“Jensen bet the company three separate times, and truly bet the company, like, if this initiative fails, it will go to zero, and it worked three separate times.”

“Every moat only works if the castle is sufficiently small. If the prize at the end of the finish line becomes sufficiently large, you're gonna need a bigger moat, and you need to figure out a you know, how to defend the castle harder.”

“Nobody is getting fired for buying Nvidia anytime soon.”

“And what these guys from Toronto did is they went out probably to their local Best Buy or equivalent in Canada. They bought two GeForce GTX-Five-Eighty's, which were the top-of-the-line cards at the time, and they wrote their algorithm, their convolutional neu…”

“Microsoft is the exclusive cloud infrastructure provider for OpenAI, which runs, as far as we know, solely on NVIDIA infrastructure, but they buy it all through Microsoft.”

“NVIDIA has literally just spent the past five years working insanely hard to build a new computing platform for The data center, a GPU accelerated computing platform to, in their minds, replace the old CPU led Intel dominated x-eighty six architecture in the d…”

“If you look at the number of CUDA developers over time, it was released in 2006, It took four years to get the first 100,000 people. Then by twenty-sixteen, 13 years in, they got to a million developers. Then just two years later, they got to two million. So 1…”

“They have 26,000 employees, and that sounds like a big number, but for comparison, Microsoft, whose market cap is only twice as big, has 220,000. So that is five X the number of employees per dollar of market cap going on over at Microsoft”

“They're actually using AI to design the chips themselves now.”

“Today there are five hundred million CUDA capable GPUs for developers to target.”

“GPUs and NVIDIA and the transformer made it possible, but to work with the size of models you're talking about spending an amount of money that's certainly for a nonprofit and anybody really except Google was untenable.”

“And by NVIDIA bifurcating their chip architectures into a gaming segment that does not have this latest COWAS technology, this allows them to monopolize, like, a huge amount of TSMC's capacity to make the COWAS chips, specifically for these H-one hundreds, whi…”

“That's gonna go away, but I don't think this very high, you know, 65% plus margin is gonna erode too much.”

“Most workloads can be run on A-Hundreds unless you're doing model training of GPT-IV.”

“So the magical unlock, of course, is to make a computer that is not a von Neumann architecture. To make programs executable in parallel and massively increase the number of processors or cores. And that is exactly what NVIDIA did on the hardware side, and all …”

“Whatever advances are happening in Moore's law and the number of transistors on a chip, If you have an algorithm that can run in parallel, which is not all problem spaces, but many can, then you can basically lever up Moore's Law by hundreds of times, or thous…”

“So starting price for DGX Cloud is 37,000 dollars a month, which will get you an A-one hundred based system, not an H-one hundred based system.”

“So they launched it in September, twenty-twenty-two. It's the successor to the A-One hundred. One GPU, one H-One hundred, cost 40,000 dollars.”

“Right now, GPUs In the data center are like a thirty billion dollar a year market going to like a fifty billion dollar next year”

“I don't think there's anyone else there where they're, like, getting ready for that person to take over. I think The company is a extension of Jensen's thoughts and will and drive and belief about the future, and that's kind of what happens.”

“I think this was one of, if not the most incredible earnings release by any scaled public company ever. Seriously, no matter what happens going forward, last week was a historic moment.”

“I think there's a pretty reasonable chance that there's some falter in the next 12 to 18 months where there's a crisis of confidence among investors where at some point something will come out Where we all observe, oh, maybe GPTs aren't as useful as we thought…”

“In one of Nvidia's earnings slides in 2021. They put up their total addressable market and they said they had a one trillion dollar TAM. And the way that they calculated this was that they were going to serve customers who Provided a hundred trillion dollars w…”

“Last year, China was 25%, or sales to mainland China, was 25% of NVIDIA's revenue.”

“So, the reason you want an H-one hundred is they're 30 times faster than an A-one hundred, which mind you is only like two and a half years older. It is nine times faster for AI training.”

“The CFO Colette Kress said on their last earnings call that about half of the revenue from the data center business unit is CSPs. And then I believe after that is the consumer internet companies, and after that is enterprises.”

“Even Google at this point in time, this is when they start building their own chips, TPUs, because, you know, they're still buying tons of hardware from NVIDIA, but they're also starting to source their own here.”

“They do make semiconductors, and they do make data center gear, but really they are a platform company. The right analogy for NVIDIA also is Microsoft. They make the operating system. They make the programming environment. They make many of the applications.”

“So I think the bottom line here is it nearly impossible to compete with them head on, and if anybody's gonna unseat NVIDIA in the future of AI and accelerated computing, it's either gonna be from some unknown flank attack that they don't see, or the future wil…”

“A listener helped us out and estimated that the cost to actually build an equivalent A-one hundred DGX system Would be today something like a 120 K. Remember, this is the previous generation. This is not H-one hundreds. And you can rent it for 37 K a month. So…”

“AMD doesn't have all this capacity reserved from TSMC, at least not for the 2.5 D packaging process for the high end GPUs. AMD doesn't have the developer ecosystem from CUDA.”

“They made one of the best acquisitions of all time back in 2020, and nobody had any idea. They bought a quirky little networking company out of Israel called Mellanox.”

“If you want to train GPT or a GPT class model, there's one option. You're doing it on NVIDIA. There's one option.”

“This last quarter, they had a gross margin of 70%, and they forecasted for next quarter to have a gross margin of 72%.”

“You can get access to a DGX server. That's eight A 100 for about 30 bucks an hour, or you can go over to AWS and get a P five dot 48 X large instance, which is eight H 100, which I believe is an HGX server for about a hundred dollars an hour.”

“They have forty six million dollars of market cap per employee.”

“Since COVID, NVIDIA has re-accelerated to a six month shipping cycle. They've been doing two GTCs a year, most years since COVID, which is insane for the level of technology complexity that they're doing.”

“They used to have to plug into other people's servers, and then they started making servers that plugged into other people's racks and rows and architectures, and then they started making their own entire rows and walls, and at some point here, they're gonna s…”

“In March of 2019, NVIDIA announced they were acquiring Mellanox for seven billion dollars in cash... In August of 2019, NVIDIA released what was at the time the largest transformer-based language model called Megatron. Eight point Three billion parameters trai…”

“NVIDIA sells these DGX systems for like, a 150 to 300,000 dollars a box.”

“And the combination of those three things turns out to be basically the single greatest moment that could ever happen for NVIDIA.”

“Up until NVIDIA's current GPU generation, the hopper generation of GPUs for the data center, there was only one GPU architecture at NVIDIA, and that same architecture and those same chips from the same wafers made at TSMC, some of them went to consumer gaming …”

“There are literally a couple of hundred million, right? 250,000,300 million installed base of active CUDA GPUs being used in the world today, and they're all architecturally compatible.”

“Your organization should be the architecture of the machinery of building the product. Right. That's what a company is.”

“The disconnection between where NVIDIA's computing is done versus where it's enjoyed, If you can separate that, your market opportunity explodes.”

“You want to pave the way to future opportunities. You can't wait until the opportunity is sitting in front of you for you to reach out for it, and so you have to anticipate. You know, our job as CEO is to look around corners and anticipate where will opportuni…”

“CUDA is used for almost all fields of science. Everything from molecular dynamics to imaging, CT reconstruction to seismic processing to, you know, weather simulations, quantum chemistry, the list goes on, right?”

“The number of smart decisions we made, which are smart to this day, how we design chips is exactly the same to this day. Because, gosh, you know, nobody's ever done it back then, and we pulled every trick in the book in a desperation because we had no other ch…”

“Now remember, at the time that we started the company in 1993, We were the only consumer, three D graphics company ever created”

“The reason why I wouldn't do it, and it goes back to why it's so hard, is building a company and building a video turned out to have been a million times harder than I expected it to be, any of us expected it to be. And at that time, if we realized the pain an…”

“Until, until I'm fired, This is my last job.”

“In a command and control system, the person who you reports to has more power than you. And the reason why they have more power than you is because they're closer to the source of information than you are. In our company, the information is disseminated fairly…”

“If he hadn't decided to go to OpenGL and accelerate, accelerate for Quake frankly, you know, what would be the killer app that put us here? And so Carmack and Sweeney, both between Unreal and Quake, created the first two killer applications for consumer three …”

“And for certain segments of the market where the technology is never good enough, like three D graphics, when we chose the right technology, three D graphics is never good enough. And we call it back then, three D gives us sustainable technology opportunity be…”

“And I said, I don't think anybody can tolerate that, but it doesn't change the fact that someday every car will be a software defined car.”

“And that time, 1997, was probably NVIDIA's best moment, and the reason for that was our backs were up against the wall, we were running out of time, we were running out of money, and for a lot of employees, running out of hope.”

“Oftentimes, if you created the market, you ended up having, you know, what people describe as moats. Because if you build your product right, and it's enabled an entire ecosystem around you, To help serve that end market, you've essentially created a platform.…”

“And so when you push your chips in, what you're really doing is you're, when you bet the farm, you're saying, I'm going to take everything in the future, all the risky things, and I'm going to pull it in advance. And that is probably the lesson. And to this da…”

“GeForce Now was NVIDIA's first data center product. And our second data center product was remote graphics, putting our GPUs in, in the world's enterprise data centers, which then led us to our third product, which combined CUDA plus our GPU, which became a su…”

“And if you just looked at the pipeline of a programmable shader, it is a processor and is highly parallel, and it is massively threaded, and it is the only processor in the world that does that.”

“We are the only accelerator on the planet where every single accelerator is architecturally compatible with the others. None has ever existed.”

“A data center is distinguished from a desktop computer versus a cell phone, not by the processor in it. A desktop computer in a data center uses the same CPUs, uses the same GPUs, apparently, right? Very close. And so it's not the chip, it's not the processing…”

“And so, that's the only unnegotiable rule in our company. Everything else is negotiable.”

“Enjoy it, be inspired by it, but not to adopt it. That's not the whole point of these books. The whole point of these books is to share their experiences, and you, you're supposed to ask, you know, what does it mean to me in my world, and what does it mean to …”

“DGX, the first version that we built was used internally, and I delivered the first one to OpenAI”

“Instead of scraping the internet we partnered with Getty and Shutterstock to create commercially fair way of applying artificial intelligence sharing to the AI.”

“You could do all the right smart things and still fail. You could do a whole bunch of dumb things, and I did many of them, and still succeed.”

“The mountain of it in the course of 31 years, if I were to take all of that all the challenges and all the hardship and all the pain and suffering of the last 32 years, and I would have compressed it into the brain of a twenty-nine-year-old, there is no way th…”

“Of course, I, it was taken out of context because I was asked about that several times after that, and of course I would start the company if I knew it would turn out this way.”

“This is almost always true, really is true in every great company that I can think of. The core senior most executive team are all people who came up in the company Or have been there for a very, very long time. It's not a whole bunch of mercenaries there at t…”

“NVIDIA was started in 1993, only ever raised twenty million dollars.”

“They make the primary chip inside every MacBook and iPhone shipped today. They're powering the AI wave, manufacturing all of NVIDIA's chips.”

“Yes. More than a hundred million. And I also said, our offer is effective, 48 hours. If you do not, there is not going to be, we are not going to argue, we are not going to bargain. If you don't accept the offer within 48 hours, we will have to go to an arbitr…”

“The previous CEO did not even give him a number, you know. The previous CEO gave him zero.”

“Within two or three years, they were, they did become one of the biggest five customers.”

“They buy two NVIDIA GeForce GTX five eighties, which were NVIDIA's top of the line gaming cards at the time. The Toronto team rewrites their neural network algorithms in CUDA, NVIDIA's programming language. They train it on these two off the shelf GTX five eig…”

“So they settle on a plan to order 40,000 GPUs. From Nvidia. For a cost of a hundred and thirty million dollars. That's a big enough price tag that the request gets elevated to Larry Page, who personally approves it, even though finance wanted to kill it, becau…”

“They're a chip company with their Tensor Processing Units, or TPUs, which is the only real scale deployment of AI chips in the world besides Nvidia GPUs.”

“But today, Google, it's estimated, has two to three million TPUs. For reference, Nvidia shipped, people don't know for sure, somewhere around four million GPUs last year.”

“Everyone is paying NVIDIA 75, 80% gross margins, implying something like a four or five X markup on what it costs to make the chips.”