The Exchanges, every show

Q What business ideas don't work in, in advertising? Like, what are the business models that just are doomed to fail?

A Trying to be a middleman on top of these large platforms. From my understanding, work, Trade Desk, I know, doesn't work on Google or Facebook at all. It doesn't work with Google or Facebook as a first party, but Applovin, I think only a little bit works on Google and Facebook. Mostly they do their stuff on the, on the unwashed web, basically, outside. So you've got to stay out of Google and Facebook's ecosystems. Because if you're trying to build your business on top of Google and Facebook, or probably soon open AI, uh, as an ad company, you're going to get squeezed. Every time you build a new capability on top of Google, in terms of Google learns what you're building, and Google has the best engineers on the planet, so do Facebook. They will take your capabilities and incorporate it into their platform. There's going to be almost certainly a cottage industry of companies that are going to come and say, I'm going to help you optimize ads in ChatGPT. There's already companies that help you optimize placement In what is called these answer engines called AEO instead of SEO. All of those are not going to create durable enduring companies.

AI assessment note: “Trying to be a middleman on top of these large platforms.”

Q What are the attributes of a good Northstar metric? Like, What advice would you give someone that's trying to pick the thing around which the company is going to optimize?

A Yeah, the North Star metric is a, is a metric that is an indicator of company growth and customer value. So it actually balances customer value and business value nicely. North Star metrics, in my opinion, should not be revenue. It should be something that is directly correlated to customer value. So for example, if customers are doing well, the North Star metric should go up and to the right, but it should also lead in business, the business doing well. For example, for Square, the North Star metric was GPV, which is Volume of Payments Processed. It was not correlated to revenue, it was somewhat correlated to revenue, but it most importantly showed that the number of, the amount of payment processed to the company was continuing to grow. At Facebook, the North Star metric was DAUs. It was actually monthly active users, then it over time went to daily active users, because it was a sense, it was an indication of how engaged users were. Now, one of the most important things about an NSM Is that it needs to be coupled with what we call check metrics. In other words, Nostra metrics, if they're let alone can, as you know, incentives drive behavior. So if you tell a team go and optimize this Nostra metric, they will do what it is going to go up a hundred percent, but then many things that you don't want to go down could go down. So for example, in, in the DoorDash case, you could sa…

AI assessment note: “a metric that is an indicator of company growth and customer value.”

Q the state of affairs? If I want to, if I want to plug one of these little batteries in, in my house, uh, maybe I'm using it as backup for my fridge or whatever it might be. Um, what is the lay of the land in terms of what requirements it needs to meet? What permissions I do need? Is it murky? Is it defined? Like, where are we there?

A It is murky. I think what really matters is the thing we focus on a lot is safety. So there are plenty of UL certified products that, that adhere to the NEC, uh, saying you can plug this in in the following manner and it's safe, uh, to do so. And so there may be jurisdictional like AHAs or DOBs or fire departments that have an opinion on what should go in a given location, how big of a battery or something like that. But at the sort of electrical code level, these are, this is already allowed under the current guidance, and there are many products that support that. So from that lens, you could say in most places you can go out and buy these things and, and plug them in in whatever state you're in. Um, a lot of the attention that's happened recently around regulations is specifically there's bills now introduced in I think it's up to 30 states, or sorry, 24 states with maybe 30 soon, um, Introducing bills where you can actually export to the grid through these devices. And so we think of that as an extremely important distinction where a lot of that regulation that's being passed is focused on really what is an interconnection agreement? What permission do I need from there for the utility? Whereas I understand the utility's concern is, hey, if you just start exporting and the grid goes down and our line workers out there, they don't actually know a line is, is energized and, a…

AI assessment note: “It is murky. I think what really matters is the thing we focus on”

Q So many generic drugs are made overseas. What, what's the biggest challenge in making them here? I mean, you, you have a facility now. It's taken you a couple years, but why is it so hard to do it in the U.S.?

A You know, that's a great question and a timely question. I'm glad you asked. Um, it's not hard for us to do because we're robotically driven. We can change the drug we're making in a four hour period, you know, so we can make a one month or one year supply of something, then roll it over to another drug. Our biggest inhibitor and why we haven't done it yet is all the fees that the FDA charges. So it's about 365,000 dollars to set up what's called an end on new drug application. And if you want to do it For a hundred drugs, that's 36 and a half million dollars. You want to do it for a thousand generics, you can do the math, right? And so because of that, that, that is what ruins the margins here in the United States. And so we can make generic pills here in the United States cheaper than what we can buy them for from India or China. We just have to deal with the associated costs of all the applications.

AI assessment note: “Our biggest inhibitor and why we haven't done it yet is all the fees”

Q Awesome. Welcome to the show, Kristen. So tell me about the origin story of the brand. How did it, how did it start?

A Yeah, so I ended up having, I had four kids in three and a half years. It was right before and at the start of COVID, and I learned firsthand how important it was to get outside for both my own sanity and for my kids' sanity, and so as winter came around, I went to get snow gear for them, and I really saw, um, a white space. I saw there's two categories of what was on the market. It was either super technical, Kept them warm and dry, but it was either just a black snowsuit or very rugged aesthetic. Or on the other bucket, you had these beautiful pieces, but they didn't actually work and wouldn't actually keep kids warm. So that's kind of when the idea sparked, and, um, I brought the two together.

AI assessment note: “So that's kind of when the idea sparked, and, um, I brought the two together.”

Q What led you to pivot from the GP side to the LP side?

A More happenstance, being lucky in terms of relationships. I was planning to stay on the GP side, but I worried about the firm where I was, great people, wonderful mentors, but was going to face these cyclical and secular declines. But I didn't believe there was a future. Our largest LP was Harvard Management Company. I went to tell the folks at HMC, who remain very good friends of mine, that I was going to leave and I'm going to stay involved with portfolio companies and sit on a couple of boards. They said, hey, would you ever think about joining us? We're going to rebuild the CoInvest platform at Harvard. The Charles Bank team had spun out already. We think you'd be a good addition to the team. That pivoted me from, I'm a GP, I'm a GP, to then thinking about being an LP. The other piece for me in that situation, Ted, I received financial aid when I went to Harvard. The mission-based aspect of that spoke to me of being able to give back in some way while doing something I really enjoyed.

AI assessment note: “They said, hey, would you ever think about joining us? We're going to rebuild”

Q business, tech, and startups. Thank you to The Hustle for sponsoring this video. Let's get back to the story. All right, Roman, I want to dive into this Reddit post you have right here, this Y Combinator post. What do you think would be, like, the top three reasons why this post went viral that people watching this can learn from and try to do with their own Reddit posts?

A Sure. So first thing would be using AI. So how I write my post is really simple. First, I'm not an English native. So what I do is I tell my story to ChatGPT by voice, and then I asked ChatGPT to translate and correct my post. So that's something that works really well because you can dive way deeper into details than if you have to write everything. Second thing why it worked is the initial start. So you need to get as many votes as you can in the first five to 10 minutes. So what I do to make it work is I have a group of friends. Where we share each other's Reddit posts, and every time someone shares the post, we all go, like, upvote, and comment. And then third thing that's really important is never putting your SaaS directly in the post, and you can trigger curiosity by highlighting it, and as I said before, putting some proof, and the people reading this post know that it's real, and that we are not telling a fake story.

AI assessment note: “So first thing would be using AI... Second thing... And then third thing”

Q really excited to talk about this, especially Reddit, because Reddit is one of my favorite channels, especially for early founders. This is going to be a fun one. But first, before we get into that, I want to understand what is the business that you built? What's the business model? And then I want to walk through some of your dashboards and show off how much this business is making.

A Uh, yeah. So we are a SaaS business. We have three types of subscription. Most of our users are on the 99 dollar plan. So you get an unlimited amount of high intent lead and you can connect them directly Through our platform. So our users get better results than using regular outreach tools. One of our main ways to get users, or at least at the beginning was Reddit. So we got eleven millions impressions on Reddit, which is insane. If we would have paid for that, that would have cost us hundreds of thousands of dollars. And so that way we got our, uh, let's say 101st customers and we are still getting customers today. So here you can see that's a 24 K MRR. So that's Around 30 K dollars and the growth is pretty good. So we are really happy with how it's going.

AI assessment note: “we are a SaaS business. We have three types of subscription.”

Q Okay, so what would be your playbook for starting over right now in 2026 in getting Reddit off the ground as a marketing channel in a way to get your first 1000 thousand users?

A So first you need a fresh account. You need one account per brother. So for example, I have one account on Chrome, one account on Mozilla, one account on Safari, and that's more than enough to keep it safe. And when you create an account, do not create it with a new email. So if you do the combo new email plus new Reddit account, You will get banned right away. Step two. Once your account is created, you have to add a profile picture. You link your SaaS in the bio, and you activate the secret feature that hides your feed. So when you do Reddit marketing and heavily post on Reddit, some people are going to go to your profile and see that you are always talking about the same SaaS, and they will say, is advertising way too much? So that's a really useful feature. Step three. For the first seven to 14 days, don't even think of doing marketing. You will just post comments. You will upvote. This will warm up your account and gets you the first karma. The more karma you have, the stronger your account is. Step four. Once you've done that, after seven to 14 days, you can start posting. And step five. Once you reach that stage, you can start doing marketing. So there are still some things to know. Some subreddits don't allow promotion. Like the Y Combinator subreddit, it's almost impossible to post anything. And so it's not worth posting there. Step six. With every post, you change the…

AI assessment note: “So first you need a fresh account. You need one account per brother.”

Q So let me push back before we go and ask you to identify any risks or threats or downside risks in all of this. What should we be worried about, if anything, with regard to AI usage?

A Well, I think there are Orwellian scenarios, uh, of AI that I think we should be concerned about, and again, I, I tend to think that those scenarios were described by George Orwell, not by, you know, James Cameron and the Terminator, and specifically, it's misuse of AI by government. I do think that AI could be used as a tool to, um, surveil, to censor, To even potentially brainwash the population. This is why the administration has taken such a firm stance against what it's called woke AI, which I almost think that that name maybe trivializes the magnitude of the problem we're talking about. We're talking about AI having a political bias built into it. Um, and the bias can be so subtle that people don't even necessarily notice over time, but it has a huge impact on what people are allowed to learn and think and know and what, you know, children learn. And so I think it's very important that we try to make sure that AI was blatantly unbiased. Um, there, just in this regard, one of the things that we were so concerned about with that by an executive order on AI that we were sending in the first week is that it had 20 pages of language on DEI, and it was promoting this idea that AI models need to build in a DEI layer. Well, you know, this is how you ended up with, you know, the, the, the Black George Washington Uh, you know, story where the, the first version of, of, uh, Gemini c…

AI assessment note: “I think there are Orwellian scenarios, uh, of AI that I think we should be concerned about”

Q Do you want to talk about, uh, mega kernels and Together Atlas for a minute?

A Together, uh, mega kernels, Together Atlas, these are both, um, projects along these lines. So let me dive into the mega kernels first. To understand this, uh, the, the first thing when we say kernels is we usually mean we are going to write a specialized GPU program for a single operation in a model. Um, you can think of a model as one of these train models is like ABC different operations in a row, and there'll be hundreds of these. And the way that we've been writing kernels for the whole history of, let's say call it NVIDIA hardware, is that you really specialize a single kernel for a single operation. With these mega kernels, we're doing something quite interesting, which is We can take the entire model, however many billions of parameters and put it into a single GPU kernel. Um, and with that, you can start to do a lot more fine grained optimization than you were able to do before. Uh, it actually starts to make the NVIDIA GPU look a little bit more like a Cerebris chip or look a little bit more like a Samba Nova chip in terms of the, the optimization that you're able to do. And this is really critical at inference time. So we're able to see two X, sometimes three X speed ups, um, over even highly optimized inference engines. Um, so we're working on bringing that to, um, uh, to, to really work in production, bring, bring it to fruition and use it across our whole stack. T…

AI assessment note: “these are both, um, projects along these lines. So let me dive into the mega kernels”

Q So in markets that are known for having so much volatility, how do you think about the process of risk management?

A One way to think about it is taking this developed market, emerging market dichotomy. In developed markets, you would say, what's my value or risk? What's one standard deviation, two standard deviations, blah, blah, blah. In emerging markets, it is more important to say, if 1998 had happened right now, what would it be? If 2008 had happened right now, what would it be? Scenario analysis is much more important than standard deviations. Because in many of those scenarios, it was a six standard deviation event. I still remember in my time at JP Morgan, I was in charge of commodities, so there was a commodity crash at some point. I was probably 36 years old. They asked me to come to the board to explain what happened. I'm sitting across from Marty Feldstein. That was already a mistake. I then tried to sound smart by saying, well, this was a Six Sigma event that happened in these markets. And Marty Feltzian without missing a beat says, so Nick, we shouldn't expect this in the next 10,000 days, right? Of course, there was another crisis like that a year later. Scenario analysis is more important than the standard deviation stuff, number one. Number two, on public markets, the most important thing from a risk management point of view is to have a currency capability, because currency liquidity almost never goes away in emerging markets. You may have an equity portfolio that is hard to…

AI assessment note: “Scenario analysis is much more important than standard deviations.”

Q As you look today at what you've done since the financial crisis, how many different acquisitions have you done in total?

A We have done around 10 or 11 in total, of which seven are still part of the business today in one form or another. The most recent one you would have read about was in Africa. Two and a half years ago, we took over a business there called Ethos Capital Partners. Very good group of people. I had known them for a long time. It's a love match. Put every effort into growing that business. It was about a 1.6 billion dollar business. One of only half a dozen surviving scale GPs in Africa. Tried four or five different ways of growing the business. Just couldn't get it done. Nobody's particular fault. I wouldn't point a finger. Africa is a tough market for raising money. It's dominated by the development finance institutions, IFC, EBRD, who are particular in the kind of managers and kind of funds they want to back. That's a different chapter entirely. Our typical commercial LPs were not that interested in Africa, so we couldn't get it done. Ultimately, rather than shrink the business to die, The partners there decided to take it back, which we thought was best for the LPs, and best for the business, and obviously, if they thought it, best for them. So that's what happened. But others mostly, if it wasn't clearly an unwind, have stayed with the firm.

AI assessment note: “We have done around 10 or 11 in total, of which seven are still part”

Q What's the breadth of capabilities you have today?

A Today, you should think of us as having three lines of business. We have a public markets line of business, private markets, and a forestry and agriculture business. In public markets, we aren't solely in listed equities, although we have a very strong background in fixed income as well. We have managed local currency debt strategies, we have managed FX strategies, and we have managed inflation-linked bond strategies in emerging markets. Today, we manage Long only equities in three single countries around the world, Turkey, Mexico, and Thailand. Our fourth invests in non-brick EM countries, so mid-sized emerging market countries and frontier, very concentrated, 20 to 30 investments at a time. Our growth is to get up and running, something we've been paper trading for a while now, which is a multi-asset class. EM long only strategy. That is the one-stop approach for emerging market investing. We also are working on a couple of other strategies. One is a carry strategy. One is a equity long short strategy. On the private market side, we invest in private credit, private equity, infrastructure, and renewables. Our biggest presence is in Latin America where we have multiple offices. And Latin America is obviously important to this administration, so I think there will be a lot of strategic capital looking at Latin America right now, for which I think we are well positioned. We have…

AI assessment note: “Today, you should think of us as having three lines of business.”

Q When you thought about how to build a horizontal emerging markets investment firm, How did you think about strategically positioning other than knowing that if you were sitting inside of a bank, you might get silos infringing on what you wanted to do?

A There are two ways to build an asset manager. One is you say, look, this is my activity. This is what I do. I'm a value investor. I'm a momentum investor. I'm a quant investor. I'm a this, I'm a that. Those of you who are interested in this, please come and talk to me. The other way is to say, look, I have these investment capabilities. How can I create a solution for you out of those investment capabilities? In a way, that's what the large asset managers do. It's a supermarket. You want a diet version, you want a full calorie version, you want the low carb version, the vegan version. We've got it all. And people poo-poo the latter. It's not true investment kind of thing. But if you want to build a big firm that is going to outlast you, I was and remain convinced that's what you have to do. I knew I was going to build a solution-oriented firm that was going to work over time backwards from what investors told us. I knew we were going to tailor it to institutional investors because I come out of JP Morgan, I know nothing else. I knew the dedication to EM was going to be the big calling card. That was the thesis. Build an infrastructure that institutional investors will be interested in. Start your first fund with enough capabilities so that you can then spin out different parts as different investors talk to you about them. And that's exactly what happened. The only fund we ever…

AI assessment note: “I knew I was going to build a solution-oriented firm that was going to work”

Q Wonderful. Ok, so, we have some really big news today that we need to address. Can you share this latest funding announcement?

A Yeah, I mean, you know, we, Zipline just closed, uh, over six hundred twenty-five million dollars, uh, which is going to accelerate our expansion both in the United States and outside the US. The big thing that happened last year is that the international part of Zipline's business, particularly the life-saving work that we do in Africa, uh, grew incredibly fast. I mean, we, uh, are now saving about 17,000 lives a year. We're gonna go from serving about 5000 hospitals and health facilities To over 20,000 hospitals and health facilities in the next 18 months. We had almost no operations in the U.S. at the beginning of last year, and we have now expanded to the point where Zipline actually does more deliveries in the U.S. than it does in the rest of the world combined. So that grew really fast, and we're expecting that business to grow, uh, by more than 10 X this year again. And so, uh, really the fundraising is designed to position Zipline so that we are ready from a capex, from a manufacturing, Uh, from an operations perspective to add a lot more metros over the coming, uh, you know, four quarters.

AI assessment note: “Zipline just closed, uh, over six hundred twenty-five million dollars”

Q You mentioned, um, OpenAI and Sam as the deal maker as well with this. Very good. I liked it. Um, I heard that you cold called Sam in the summer of twenty-twenty-two. Can you just tell me about that before we do it? Yeah.

A So it wasn't a cold call. We, we cold emailed them, um, and we cold emailed Sam and Jason Kwan. Um, what we had basically done was we went on r slash legal advice, which is basically like a subreddit for asking legal questions. And we grabbed a bunch of those questions, ran a chain of thought product that we had basically built on top of it and gave it to a bunch of landlord tenant. Attorneys. And then we basically said, just like, look at these questions and tell me if they're, you would send the answer. We didn't say anything about AI to the consumer who asked the question. And of 86 out of a hundred questions, three out of three said, this is a perfect answer. I'm done. And we cobbled all that together and we just sent a cold email to Sam Altman and Jason Kwan. Um, the idea was basically, hey, did you guys know that, you know, at this point it was GPT-III and just the API was public. I think The end of 2021 or beginning of 2022, they had an API. Um, did you know it was this good at legal? That was it. That was basically the subject line of the email was like, did you know it was this good at legal? Um, and we met them like pretty recently after that.

AI assessment note: “So it wasn't a cold call. We, we cold emailed them”

Q On the margin, what's an example of something you looked at differently because of that dynamic where so much of the capital was on your own balance sheet?

A Up until twenty-twenty-two, from 2010 to twenty-twenty-two, risk-free rate went to zero and basically stayed there. That led to a risking of investors. If you had a fixed return you had to achieve, you couldn't get there in the old way of investing. You had to keep creeping up The risk curve. We'd said that's not always right. One example was the high yield market. In December of 2021, the high yield index was four and a half percent. When I started doing buyouts 30 years ago, if I got my bond deal done inside of 12%, I considered that a good day. At four and a half percent for junior capital in a levered capital structure, that wasn't good risk return. If you looked at our entire footprint at the time, we had virtually no high yield on the Apollo platform. Now, could we have gone out and raised high yield funds? Yeah, absolutely. But it wasn't the right risk return. Similarly, the real estate market. Over the last 40 years, commercial real estate had basically gotten ground down to the point of being a proxy for IG bonds. In 2021, the cap rate on any commercial real estate asset It was probably three percent. Things were getting priced in the twos. So we're sitting here at Nine West. Right behind us is the Plaza Hotel. I remember when that was being sold, we could have bought that at a three and a half percent cap rate for the equity of a hotel that needed a turnaround, or I c…

AI assessment note: “One example was the high yield market. In December of 2021, the high yield index”

Q So after that period of time, the firm is effectively a boutique private equity firm, sounds like, series of funds. When did you start to evolve and say, we could do something more, something bigger?

A That was the GFC, the financial crisis. The financial crisis opened our eyes to a lot of opportunities. One, we had raised a fund right at the beginning of oh eight. So an unbelievable opportunity to deploy capital at Either good valuations or in distress situations where you could buy amazing companies, companies that Apollo never could have acquired at unbelievable valuations. That was a real game changer for our private equity business, but really the culmination of all the hard work over the prior 1520 years. It also opened up a couple of things. When I said how we think about investing in different parts of the capital structure, as the whole Financial systems started coming unglued. Banks wouldn't lend to other banks. The ability to obtain liquidity became problematic for companies, for banks, for other things. We were able to approach banks and buy tens of billions of bank debt at a time at deeply discounted prices. We started accumulating enormous amounts of corporate debt. Not all of it was distressed. It was just the seller was freaking out. The markets were freaking out. So we're buying good paper at discounted prices. It That moment, it became clear to us that the provision of capital to levered companies is the other side of the coin of providing equity in levered situations. Private credit and private equity were two sides of the same coin. We were the first folks…

AI assessment note: “That was the GFC, the financial crisis. The financial crisis opened our eyes”

Q So if we take a step back from the evolution of the products over time, I'd love to dive into your roles. Going from a dealmaker to a leader of the business. At what point in time did you leading the teams at Apollo and working on all these strategic initiatives compared to the day-to-day dealmaking?

A After the GFC, as Apollo and as me personally did some of the best deals that I think we've ever done as a firm, I guess it was about 2010 the founders asked me to become lead partner for private equity. The firm was starting to grow for the first time into these other areas. Founders were spending more time in other parts of the business. For the first time, the PE business needed a leader other than the founders. That was my reluctant first step into the land of management. I was able to be a player coach at the time, still one leg in the deal business, one leg in the leadership business. And I played that role until about 20 18. So from 20 11 to 20 18. At the end of 2018, the firm had continued to grow and scale in a way when myself and one of my colleagues, Jim Zelter, we were elevated to co-president across the whole firm, looking after all of our revenue generating businesses.

AI assessment note: “I guess it was about 2010 the founders asked me to become lead partner”

Q to switch topics now to the growth because everyone knows who's watching this. You'll read all the comments. They'll say, okay, you don't talk about the marketing enough. Building stuff is easy now, but as I understand, you were the builder. And the marketer and the growth person. You did all this by yourself. What I want to understand is how did you actually grow this thing and get users?

A In the beginning, I didn't know what worked. So I literally just tried a bunch of different formats and I started posting around 40 times a day on 12 different accounts. Volume really matters here. What ended up working for me was the classic, you know, GDC girl reaction video with a demo on it. That's what one of my biggest competitors was doing and they were just killing it with it. I was broke so I couldn't pay For clips, I couldn't pay for another creator, so I just told my girlfriend at the time, now wife, to help me record some reactions herself, and she honestly killed it. We started posting twice a day. It wasn't until after 40 posts that I finally found a hook that worked, and I knew I just needed to spam that in both TikTok and Instagram. When you start seeing that a format starts to work, put all your energies into that. That alone, with only that account, I was getting around 10 K views a day, which was all I needed To get to my first one K a month. With that, I immediately reached out to my friends and I wanted to scale this and I hired another UGC creators. It was honestly very hard to get her account. This other creators account going on both tick tock and Insta. But after 14 days, we finally cracked one video, which organically got around 20 K views. Knowing how views convert to downloads is very tricky. In my case, 10 K views A day was getting me around two to …

AI assessment note: “What ended up working for me was the classic, you know, GDC girl reaction video”

Q to get into all how you built this, grew this, and even a little bit of paid ads, which I'm excited about. But before, I know you started it this year, which is insane that it's grown this much. I'd love for you to give me a little bit of background on how you get here, how you decide to start building apps. How do you get to this point?

A I actually don't come from a tech background. I actually majored in finance, but I discovered the original Codex model back in 2022. When I saw that, I just completely fell in love with AI and technology, and I decided that's what I wanted to do for the rest of my life. So I taught myself about AI, about how to code. I've been building tech and AI businesses ever since. Back in January of 2025, I was actually building an adult content AI app, but when I finished it, I just felt completely like disgusted and disappointed with myself. So I decided to make the weirdest pivot in history and turn into the Christian space to try to reconnect with my faith. And after about three months of building apps, nobody wanted, I came across this app called pray screen that blocked your phone until you prayed. And with the new insights I had, I looked at the design. I looked at the features, the way they were executing their distribution. And I said, you know what? I can do a better job than them. And so that's how prayer lock came to life.

AI assessment note: “I majored in finance, but I discovered the original Codex model back in 2022.”

Q Great. We'll go into, um, all of this in a, in a minute, but before doing so, uh, we, we alluded to, to some of your background. Let's go into it. Starting from the, from the beginning, what was your path to becoming a top researcher?

A I grew up in Russia, in Moscow. Starting from like middle school, high school, I was really interested in mathematics, and I was thinking I will be a mathematician or, or engineer of some kind. I was interested in machines, uh, and, and eventually, uh, computers, and I got into an undergrad in computer science, and I was still thinking that I'll be doing some kind of theoretical, you know, applied linear algebra, tensor methods, things like that. Uh, but at some point, I, Kind of discovered, uh, machine learning. There was this, uh, professor that we had, uh, Dmitry Vetrov, who had one of the, like, leading labs in machine learning in Russia at the time, and I was lucky enough to join that lab and start doing some research on machine learning in my undergrad. So that was around 2013, maybe 2014. I initially was working on non-neural network machine learning, uh, methods, so Gaussian processes. That's kind of By now, you know, nobody really talks about that anymore, but eventually I, I got into a PhD thinking I would still be doing a Gaussian process, but I ended up working on deep learning, and that was actually quite, I'm happy that I didn't work on Gaussian process. I worked on some things related to kind of core machine learning, methodology, optimization, probabilistic methods, questions related to generalization and how the models learn features. After I finished my PhD, I…

AI assessment note: “I was lucky enough to join that lab and start doing some research on machine learning”

Q And then, uh, within that world, your work specifically has focused on weak to strong. Uh, can you explain what that is?

A That is, uh, the project that we did, uh, back at OpenAI. That Work was focusing on the future scenario when we will be trying to align models that are above our own capability on certain tasks. Already now, if you take the frontier LLMs, they are extremely capable, and on a lot of domains, we need expert humans to be able to tell which responses are good, which are correct, which are not correct. But in the future, we are imagining we will have models that are More capable than humans, and even expert humans will not be able to reliably grade very complicated answers from the model. So imagine you ask it to make a repo for you for some, like, you know, new startup idea and just implement it from scratch entirely, and then it gives you, you know, 10,000 lines of code. You have no way of checking if all of this code is correct, if all of this code is safe to use. And so that's the problem of supervision. We are moving to this future when Like it's very hard for a human to supervise, uh, the models, uh, directly. And so instead we studied a simplified setting where we used a small model to try to supervise a larger model.

AI assessment note: “we used a small model to try to supervise a larger model”

Q So, uh, genai.mil, I think was like a surprise and how quickly, um, it happened. Uh, uh, can you tell us the story of that and what the, like the goal is?

A So genai.mil, because if you're on a, on a, Department of War network. You, you know, even the unclassified networks are secure, right? And, and different levels of security as you go up. So you still have to figure out how to architect using AI in, into those networks. It's not as simple as going to chatgbt.com and, and signing up because those things are sort of generally prohibited because you have to have restrictions on how to use them. So we had to figure out what are the, the policies, how do you architect it in the network? So nothing goes Back into the pool of data that ChatGPT or Claude or any of these companies have, because we certainly don't want, no one in this country wants our data getting out into the general public, right? Into these models. So you have to architect a different, uh, data flow. So, but we moved really fast. I had some like Databricks engineers, former Databricks engineers, former Meta engineers, former AWS people, Tiger team, 60 days, Uh, good, great collaboration from Gemini, uh, from Google's Gemini, who already works at Department of Warsets, some familiarity with our architecture and our systems, and got that launched to three million people. And we've had over a million people Unix use it in the last 30 days, which is kind of awesome. We've got one third of the enterprise on one model. That's 2.5, not three point oh, three point oh is comi…

AI assessment note: “Tiger team, 60 days, Uh, good, great collaboration from Gemini”

Q for or procured. And, you know, that's somewhat different from how you think about innovation in terms of something new comes up. You want to rapidly be able to iterate, to launch it, to deploy it. And so if, how do you think about either flexible budget spend or reallocation of budget over time and how does that relate to the legislative process and how does that impact entrepreneurs ultimately?

A Yeah, it's been, it's been a problem, such a problem that they've given like an awful name to it called the valley of death, which I, which, uh, you'll hear eight ways to Sunday here. And the ways, you know, we've come at it and I've come at it is we have this defense innovation unit, which is rapid contracting, has a billion dollars, has like a reasonable amount of money to do things really fast to get startups off the ground and get them through. We have several other programs that have that same capability, one called AFIT that takes companies that have developed the product, but now need to scale their manufacturing. So there's a different line in the, you know, different point in the process. I have the Office of Strategic Capital, which has two hundred billion dollars in lending authority, um, low cost loans, and that both sends a demand signal to these companies that other private capital crowds around it, equity capital often, and it's, Low cost loans, right? It's treasuries plus a hundred bips. And in the last, I don't know, four or five months, we've done five critical minerals deals, like really fast. Cause that's an, that's a target area and we'll have other target areas. So if you're a company that's in one of the super critical areas, um, we have this huge lending authority there too. So I'm trying to collapse the valley of death by crowding capital around in diff…

AI assessment note: “we're reserving a bit of the budget every year so that we can make in-year budget decisions.”

Q Alright, Cowboy Country Club, a made-up golf club, and, and you sell hats and apparel, and tell me the story about that, like, what, how did this happen?

A Absolutely. So, growing up, my parents were members of a golf club, uh, so I had a community to play with and talk about golf with, and then I went off to college, had a great community there to play with, then after graduation, I went and took a finance job in Atlanta, and being there, I didn't have anyone to play with, I felt excluded from the game, nowhere to call home, and I figured that I could solve my own problem of feeling excluded by creating my own country club and inviting people to join. It, it was a crazy idea, but we built a social media following where I told my story and talked about why this was something that I was so passionate about, and our first drop sold out in 12 seconds.

AI assessment note: “I figured that I could solve my own problem of feeling excluded by creating my own”

Q And so how does that, so you get 3800 some odd applications. How does that translate into the selection committee making decisions as to which founders to bring into a batch?

A Right. So we use humans so far, and we'll continue to use humans. I'm sure there'll be some AI in there, but so as I mentioned, you know, we have alums, Berkeley alums, who are very excited about helping us and other people, not just Berkeley alums, but anybody who goes, oh, this is, this is a cool place to be. I like helping startups. So we have about 900 people Mostly Berkeley alums, but not all who have signed an agreement to be advisors for our startups pro bono, no charge. We don't pay them and they don't charge the startups anything when the startups are in the program. So of those 900, a few hundred are part of our selection committee. So we divide up all those applications by industries. We assign them to advisors based on their expertise and we say help us score them, uh, online. And so from that big 3800, we narrow that down to about 170 for a first round interview. And again, we invite advisors based on expertise. Uh, sometimes we'll invite Berkeley faculty if it's highly technical and we really need some deep technical expertise. And then from that, we narrow that down to a set of finalists of about 70. And that's a longer interview with even more due diligence. And voila. 20 or so companies, um, are selected for the accelerator batch.

AI assessment note: “from that big 3800, we narrow that down to about 170 for a first”

Q Wow. Um, what is, uh, what is international response been to the program?

A It's been huge. So we are a university program that doesn't teach or do research. So like any one of those programs on campus, we have to raise our own funds. So we do that primarily through international partnerships. So our main source of revenue to support the program is partnerships with mainly international governments. So our biggest partner is JETRO, the Japanese External Trade Association. They send startups to participate in our program. They don't get funding. They're not accelerated, but we have a special program for them. And they pay us a sponsorship fee per startup. And then more and more we were being asked to go to other countries to do boot camps and workshops and bring our learnings to the startups in their home country.

AI assessment note: “It's been huge. So we do that primarily through international partnerships.”

Q How did you find the restaurants? Um, how did you get into them? How did you, did you just kind of pound the pavement and start to call and, and.

A Well, I'll tell you about the first one. And then that started the, the whole kind of Process. So it was October 15th. We walked on the street. We went to the number one restaurant in the country at the time, Le Becfant. We went into the back of the kitchen. Someone stopped and asked us, what are you doing here? And we said, we're here to make coffee for chef. So they just assumed maybe that was an interview. So we went, we tore the grinder down, cleaned it, tore the espresso machine, cleaned that, put our coffee into it, made a beautiful, beautiful coffee, and I said, where's chef? We went upstairs. He was screaming as, you know, he was a screamer. It was the nineties, right? Wanted to know. And we said, we're here to make coffee for you. And we just put it in front of him and you just step back. And so that's how we got the Becfant that day.

AI assessment note: “We walked on the street. We went to the number one restaurant”