The Exchanges

Q You said about the inherent risks that you take on when you start a company, obviously. Do you believe in the advice that you should only pursue ideas that only you can do? In other words, the idea is specifically tailored to you and not everyone could solve that problem? Or do you think that's bullshit and it's actually about execution?

A I actually do believe in it. So again, if you think about the idea of a startup as something that is a place where you can take big risks, where you can build something that nobody else can, And you're willing to just go all out to, again, create something that literally nobody else could. It does have to be something unique to you, because again, like, otherwise you're like, sure, you can get to like a, like a decent medium-sized company with a commodity idea, but if you really want to go big, if you really want to build a generational foundational company, I think it really should be about an, About an idea that is almost like unique to you.

AI assessment note: “I actually do believe in it.”

Q It is 2040, and we still do not have AGI. What is the primary reason why that would be the case?

A So I think there are two reasons. One is that there will always need to be more breakthroughs, whether it's breakthroughs, and yeah, how you leverage all this data, or breakthroughs in different types of algorithms that you're, that you're, that you're building. And then another one is just how you gather that data. Like, at the end of the day, I think a lot of data will be, it's like, in order to cure cancer, how will you Gather the data that's needed to make those breakthroughs. Maybe you're gonna have to run real-world experiments, real-world studies, and those studies will simply take time. And so is there, will there be a way to speed up those experiments through various kinds of simulations or just other forms of gathering data? Um, I, I, I don't know, but there is Did somebody question how do you get the data even faster? Which I, which I think will be very, very important.

AI assessment note: “So I think there are two reasons. One is that there will always need to be more breakthroughs”

Q I spoke to Garrett at Handshake right after the acquisition. He said, like, I'm just staying up all night. There's just a tidal wave of scale customers moving to us. Did you have the same though in terms of that tidal shift in customer demand shifting to you as well as the realization that you mentioned there?

A Yep. I mean, so I would say I'm pretty sure that a lot of these other companies, they are, um, Like at the end of the day, people want high quality data and they don't want to be working with body shops. And so. I think we've seen, like, a massive wave interest because, like, yeah, like, the space is really large, and there are a lot of teams who are still using scale for legacy reasons. It's like, at the end of the day, we were already the biggest and best in this space, and so even when there were teams at some of these larger companies who weren't working with us already, they, like, they kind of, like, knew who to turn to.

AI assessment note: “Yep... I think we've seen, like, a massive wave interest”

Q You said about the inherent risks that you take on when you start a company, obviously. Do you believe in the advice that you should only pursue ideas that only you can do? In other words, the idea is specifically tailored to you and not everyone could solve that problem? Or do you think that's bullshit and it's actually about execution?

A I actually do believe in it. So again, if you think about the idea of a startup as something that is a place where you can take big risks, where you can build something that nobody else can, And you're willing to just go all out to, again, create something that literally nobody else could. It does have to be something unique to you, because again, like, otherwise you're like, sure, you can get to like a, like a decent medium-sized company with a commodity idea, but if you really want to go big, if you really want to build a generational foundational company, I think it really should be about an, About an idea that is almost like unique to you.

AI assessment note: “I actually do believe in it.”

Q Is there a time when you let quality slip in any area of the company? And with hindsight, what did you learn from that?

A I, I think we've, we've never let quality slip. I mean, it's so, it's such a principle ingrained to, into everybody at the company. Like one of the things that we simply tell everybody when we first join, quality is the most important thing. It's more important than, uh, yeah, it's more important than anything else. If you have to make a deadline slip, because for whatever reason, you don't think the quality is there. If we have to say no to a project because We just can't handle it right now. I mean, uh, we can generally handle a lot of things, but for like, we, we just want to ingrain this principle that is okay to say no. It is okay to, um, kind of like let other things maybe slip just because we, we care about quality at the end of the day.

AI assessment note: “I, I think we've, we've never let quality slip.”

Q but they're bluntly, they seemingly are much more behind because they've built their own models and they believe very much in the power of verticalization of models and specific models, uh, or specialized models, so to speak. How do you think about the future in terms of monolithic, generalized, very large scale models versus the requirement to have very narrow, very specialized models For things like code creation and development?

A So I think there's an opportunity for both, and the reason I think that is, it's because on the one hand you have these giant, all-powerful models, and sure, they can be really, really good and really, really powerful, and in like a raw capability sense, At least right now, I think they will be able to do what they'll be, they'll be able to encompass all of these different use cases, but in the same way that a company, so take a company like Google or Facebook, there are simply some products that they can't build because building those products would be counter to Like culture or the business goals of like the overall parent company. And so in the same way, sometimes you need to be able to move faster and to take big bets on certain kinds of products. And like the, like the all powerful model just can't kind of let that happen because if you let it happen within this, like one small domain, we'll kind of almost like pervade the entire model. So sometimes you do need to like the smaller models to, to break through if they have like a really unique view on how, on how you're operating.

AI assessment note: “So I think there's an opportunity for both, and the reason I think that is”

Q Do you think you have a fundamentally different business then? Because you're all lumped in the same category. But if they're passing along a warm body, and you're passing along data, it's a phenomenally different product, and it's monetized differently, no?

A Yep, yep. Yeah, again, like, if I think about the way we think about it, it's, it's maybe the following. Like, so we have always started out with quality of the data as our number one principle. And as a result, we need to build a lot of technology in order to, to measure that and improve that. If I think about like what, what goes wrong, it's that people often just don't realize how difficult quality control is. And people often think that humans are smart. And so if you just throw a bunch of humans out of the problem, you'll get good data. And what we found is that's, is that, that, that, that is completely untrue. Like for example, I went to MIT, but yeah, I think half of the people who graduate with a CS degree, they, they can't even code. So it's a really challenging problem to detect high quality. And second, if you actually take the folks from MIT who can code, they're actually just going to try to cheat you. They're going to sell their accounts to somebody in a third world country. They're going to try to use LLMs to generate the data for you. They're going to come up with all these crazy methods to cheat a system. So it's also this really, really challenging problem to detect low quality. It's actually really adversarial. And so what we found is that when you want to get the highest quality data to train LLMs that are already, you know, super intelligent, you actually …

AI assessment note: “Can't just take warm bodies or... throw people at the problem and get good”

Q Can I ask, like, prioritization is slightly ambiguous according to different people. Everyone feels that their project is important and more important than someone else's. How do you determine priorities within a company and determine what matters versus what doesn't?

A Yeah, I mean, I think a big thing about being nice is that, or being small, is that When you're smaller, that means that I, other people around the company, we just have a much better view into the customer problems themselves and what everybody's working on. And so it's kind of like at, at these bigger companies, a lot of your priorities, a lot of the things that you're building, they're simply, you're simply building them to impress someone. Like, hey, I need to impress my VP. I need to impress my manager. I need to impress my director so that I can get promoted. And you're not really building things or prioritizing things because they're good for the End customer. They're good for the end product. It's more like, okay, I have this priority to, like, let me think about it. It's like, I have this priority to improve an internal tool. Okay, why are you improving the internal tool? Well, it will make people five percent more productive. Why do I want them to be five percent more productive? Because they're spending 10, 20% of their time interviewing. Why are they interviewing? Because they, like, they're growing for the sake of growing, and it just, like, leads to this perpetual cycle where A lot of your priorities are just divorced from, um, from like the end, end customer, the end product, and they're almost like priorities just for the sake of internal company machinery. So y…

AI assessment note: “we just have a much better view into the customer problems themselves”

Q Sorry, 2020, automated job of the average engineer. I had Vlad on the show from Robin Hood today, it went out, and he said 50% of code created by Robin Hood is now by AI. Benioff said the same on the show, 50%. Are we not at that stage already? How much code from Surge is created with AI?

A I don't think we're at that stage yet. At least if you're, again, if you're, if you're working on deeper problems that aren't just random features, like again, if you're Concentrating your company on the 10% of problems that are most important. I don't think models today can write 50% of the code, um, and, you know, come up with 50% of the ideas that, that people are, are, that are actually going to be meaningful to your company. Sure, if 90% of your company is writing little features that nobody cares about, or improving the, the efficiency of your code base by one percent, then yeah. But, uh, I, I don't think we're at a point if you're, if you're really working on meaningful problems.

AI assessment note: “I don't think we're at that stage yet.”

Q Is there anything that you think Elon does specifically to inspire his team to have that form of culture when they're not a small company?

A I think it's almost that you know what you're getting into when you, when you, when you work at Grok, or when you work at XAI, or any of these other companies. Like, you know, when you interview Dad, these people are incredibly mission-oriented. You know, when you interview Dad, everybody works super hard. You know that if you want, To work there, you're going to have to be the kind of person who has the same values. Otherwise you just shouldn't, you just shouldn't enjoy it because you'll be miserable. And so, um, it's this, this fact that it has such a strong culture and such a strong belief in what you're doing. It just attracts people of some, some more talent.

AI assessment note: “you know what you're getting into when you, when you, when you work at Grok”

Q Okay. Love that. So post chat GPT, you really see the inflection point. Another one that I guess is probably quite an important one is scale of selling and the movement of customers away. How did the world change for you with the scale acquisition?

A So it's interesting because I think it was an open secret where a lot of top researchers already knew who we were. And they already knew that we were the biggest and the best in the space, even though we've been pretty under the radar. And so most people were already working with us. And so, yeah, there were, there are a lot of teams who are using scale for legacy reasons, or they just didn't happen to know about us. So we've been getting a lot of new interest for them too. I think the more interesting thing has been, it's kind of been really fun seeing how we've opened their eyes to what really amazing, really high quality data can actually look like. Like a lot of them have tried getting human data from other teams and they tell us it's been this slog. They'll spend months trying to improve the data quality for really basic stuff. And it will look like it's better for a month, but then it will just quickly progress. And so we have this concept where we just want to get started immediately. We want to show them really, really high quality data immediately. But then we also want to like, one of the big concepts for us as a company is We always want to be producing data that is That you simply couldn't get anywhere else. Like, there's so much richness and complexity in types of things that we do that we just want to open up new avenues of research and open up new avenues of, lik…

AI assessment note: “we've been getting a lot of new interest for them too”

Q What single metric defines the health of the business to you? What metric, if I showed it to you every morning, you'd be like, okay, I know the state of my business.

A If I could paint my perfect North Star, and this is something that I think we want to work towards, like, it's something that we actually, we actually want to build for the industry, but it's like, Are models progressing in fundamental ways? Like are they actually getting more intelligent? Like are their capabilities improving again, as opposed to simply climbing up a meaningless clickbait leaderboard. So are these models progressing? And then how much of that is kind of like due to us, whether it's due to our training data or whether it's due to the evaluations we provide or whether it's due to the insights that we provide, um, Provide all these researchers for, for ways that they can improve their models. Like if there are a way to measure that, I would love it. Um, I think that the closest proxy we have for it today is just like the, the variety and the variety of projects that we're creating. Because I think one of the things I, again, one of the things I really, really believe in is we want to make it easy for all of these researchers to come up with new ideas. And to not be blocked by data. So the more complex, the more diverse, the more creative projects that we can provide, like, that, that, that is, like, almost a proxy for, for, for that overall store.

AI assessment note: “the closest proxy we have for it today is just like the, the variety”

Q How do you think about what you just said there in terms of building with your customers, being so close to them, letting you shape your product? But then also not doing the Henry Ford of building a faster horse, and then also not building a product that bluntly isn't relevant for a wider audience base, and you really just kind of tie yourself into a few small clients.

A So I think this is where we actually have a really, very strong vision of what a product should be. So again, like going back to what I said earlier about how most companies in this space actually don't have any conception of, I mean, both within our space, but maybe also at large, they don't have product principles that they try to adhere to. Again, like we had very strong product principles from a start. We wanted to focus on quality above all else. Like if whoever thought that we couldn't give the quality that we, that we wanted, we would just say no. As opposed to these other companies where they're almost like desperate and racing around just trying to get any traction that they can. They're trying to prove to the recs that their numbers are always going up. They're almost like focused on getting 10 dollars, a hundred dollars, a thousand dollars, wherever they can. And so as soon as some customer comes to them, even if that customer is counter to the kind of product that they're building, if they're offering money, they'll just say, sure. I'll, I'll do it just because they'll give me another logo from a website. They'll give me another case study to show another customer. They'll give me another talking point with my VCs. Like we just, I think we're very lucky to not have to worry about that because we could build for the long-term vision we had as opposed to again, like a…

AI assessment note: “we had very strong product principles from a start... we would just say no”

Q And so, in the early days, everyone else is acquiring supply side of talent, correct? All the other people that compete in the space. And you're not acquiring that talent supply, you're building product, correct?

A Um, I mean, it was both because, I mean, obviously we need a talent supply in order to make our product work, but again, it was less about, so, so there are some companies in this space who will simply think of it as a pure supply problem, and they don't give any consideration to the technology, like both the technology, the underlying technology, like, how do you identify these people? How do you make sure that they're doing good work? How do you remove the bad quality work? Like, they're just literally not They're not thinking about any of the technology aspects at all. And they're also not thinking about the product at all. Like how do you present the data to the customers? Like one of our principles, like one of the principles that I've always had, even prior to surge when I was just an ML engineer or data scientist, one of the things that I've always tried to encourage is what we call this visceral understanding of the data. Like I really just want you to go in and get your hands dirty and look at the data. Like historically, a lot of ML engineers, they kind of just don't take the time to look at the data. And maybe that's because the data just isn't all that interesting. Like, when all you're doing is drawing bounding boxes on cars, sure, I don't need to look at a thousand bounding boxes, but when you're doing is, yeah, creating poetry, creating mathematical equations, cr…

AI assessment note: “it was both because, I mean, obviously we need a talent supply”

Q Totally get you. Were you surprised By how far Elon has been able to get with Grok as fast as he has done or not? Like again, I think Elon has this, it's kind of funny.

A So before we worked with the team, I didn't really have a conception of what an Elon company was like, but yeah, I mean, we, we work really closely with the XAI team and it's actually just incredibly refreshing to see how they operate. Like they are all Very, very mission-oriented, and they're all incredibly smart, and they work incredibly hard. Like, it will be 11 p.m. at night, and I'll, I'll DM them, and someone will want to jump on a meeting, and yeah, I jump on a meeting with them, and I see them. They're in the office, and they're, there's a ton of people behind them, so like, they're all, they're just like crazy hacking together on all these problems, and so I actually think it's incredible, and it's this Kind of embodiment of what a startup can do when you really believe in something and are kind of like willing to do whatever it takes to achieve it, as opposed to living within the confines of this giant bureaucracy. So I, I think it's actually really, really impressive.

AI assessment note: “I didn't really have a conception of what an Elon company was like”

Q If that's the landscape today, which is PhDs aren't enough, a lot of PhDs aren't great quality, how does that change over time? Will you have a dramatically larger supply side? How will the tooling of the supply side change? How will their ability to turn around work change?

A Ok, yeah, so, Again, I think this boils down to the technology that we build. Like over time, you're, it's simply true that people are going to be trying to solve more and more problems. And so when you have like us, like we have hundreds of thousands, millions of people working on our platform. When you do that and you have a thousand projects, like 10,000 projects that are literally running in any given week. How do you make sure that you are building technology to identify who are the top one percent, top two percent of people who can really push the boundaries of physical problems with these models? Or how do you identify the top, again, like two or three percent of people who are writing the most amazing poetry? How do you find those people? And then also how do you remove the worst of the worst? The people who inevitably tried to cheat you and spam you, and they will basically regress the models if you allowed their data through. Like it actually is a really, really profound problem, and you just need a lot of technology to build this. And then at the same time, like these are researchers who want to move really fast. Like researchers to all of these frontier labs, they're, again, like all the algorithms are changing every day. And so they want to learn, they want to try out new projects every single week. And so if you're not moving fast enough, like if you're unable to …

AI assessment note: “this boils down to the technology that we build”

Q Speaking of kind of evolutions with AGI there, I do just want to ask on like the changing nature of data. How will the data needed evolve as AI gets smarter and smarter and smarter with each evolution?

A So a lot of people talk about the shift to BHC level data. And yeah, I think it's important. Like, yeah, it's actually really interesting how we, like, we basically have the biggest group of the smartest people in the world working on a platform. Like we actually have Harvard professors and Stanford PhD students and Princeton computer science theorists working on all these really interesting problems with us. It's kind of crazy. Like if you think of all the PhDs, even at Google or Meta or Microsoft, we have way more than all of them combined doing work for us in a single day. And it's also, they're not just writing random JavaScript code to improve ads. They're actually pushing the frontiers of science when they're collaborating with these models. But I think what people underestimate is that having a PhD isn't enough. Like a lot of PhDs, they just aren't good at this type of work. Like, again, like I said before, there are a lot of body shops and recruiting shops in our space that basically just look whether you wrote down that you have a PhD on your resume, and they'll just instantly give you work if so. But a lot of PhDs just aren't very good. Like, I think 80% of the computer science PhDs I know, they write shitty code because they're only good at math and algorithms. And then you think about people like Ernest Hemingway. He didn't have a PhD. I don't, I don't think he even…

AI assessment note: “we basically have the biggest group of the smartest people in the world”

Q How so, how does that look? Because, like, when you think about, like, funding them, I, the capital intensity or capital requirements are so large. I don't know anyone who's willing to. All the big players in the financing world, bluntly, have already got their horses in terms of this race. How does that even work?

A I think it's because it depends on what you view The long-term vision for AGI to be like, if you believe that we are still only like, despite all of the immense progress that we've made, if you believe that there's only, we're only like, I don't know, one percent, five percent of the way towards AGI, because yeah, we literally want AGI systems that can in the future cure cancer and send rocket ships to Mars and like design entirely new philosophical systems. Like these are big, massive Problems. And as opposed to simply automating away the job of the average L three or L four software engineer is if you believe that there's like, we're only, again, like only two percent or five percent of the way there, there's so much more headroom. It's like, almost like asking, do you believe, you know, 10 years ago that Google was going to be the final search engine in the world? Like sure, if you, if you like only looking forward to that, like the next five years, moving towards the, the span in some sense of like, if you just think of the immensity of what AGI could do to just like, there's so much more ahead of us than behind us that there could be these serendipitous, um, like very, very creative breakthroughs that just nobody's expecting. In part because, like, maybe it's going to be created by, by some of the AIs themselves or AIs in concert with humans. There's just so much, so much …

AI assessment note: “depends on what you view The long-term vision for AGI to be like”