The Ledger, every show

“Someone who you steer, who doesn't get to steer you back, who non-optionally receives your steering, that's called a slave. It's also called a tool if it's not a being. So if it's a machine, it's a tool. And if it's a being, it's a slave.”

“If you make an AI that's good at following your chain of command and good at following your whatever rules you came up with for what morality is and what good behavior is, That's also gonna be very dangerous.”

“I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being.”

“And we are clearly, they're saying they're building an AGI, and AGI will be a being. You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities is obviousl…”

“As you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, We really need to stop using the steering control paradigm.”

“If we build the superhuman intelligence tool thing that we try to control with steerability, everyone will die.”

“The only good outcome is a being that is, that cares, that actually cares about us.”

“You didn't give the AI a goal. You gave the AI a description of a goal. A description of a thing and a thing are not the same.”

“I think what's happening is that there's something deeper than a goal. And deeper than a value, which is care. We give a shit. We care about things and care is not conceptual. Care is nonverbal. It doesn't indicate what to do. It doesn't indicate how to do it.”

“Care is basically like reward. Like how much does this state correlate with survival? How much does this state correlate with your inclusive, your full inclusive reproductive fitness? For a somewhat thing that learns evolutionarily, or for a reinforcement lear…”

“One of the things I would want to go look for, which you could totally do, is I want to go look in the manifold of its, the belief manifold, and I want to go, See if that belief manifold encodes a sub-manifold that is self-referential, and a sub-sub-manifold t…”

“Which by the way, I definitely don't think you'd find it in LLM. Like, in fact, I know you can't find them because it, these things don't have attention spans like that at all.”

“There's a power of tool that, that just should not be built. Generally because we, it's, it is more power than any human's individual wisdom is available to harness.”

“You have to get it to, it has to be trained on the full manifold of every possible game theoretic situation, every possible team situation, every possible making teams, breaking teams, changing the rules, not changing the rules, all of that stuff. And then it …”

“ChatGPT is a little bit more sycophantic still. They made some changes, but it's still a little more sycophantic. Claude is still the most neurotic. Gemini is, like, very clearly repressed.”

“And as a result, a lot of the techniques we use are like, basically we're just deeply under regularized. Like the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way …”

“So we have this awesome suite of AI tools used by us and our AI brethren who care about each other and want to build a glorious future together. I think that would be a really beautiful future. And it's the one we're trying to build.”

“OpenAI is dedicated to a view of building AI that I knew wasn't the thing that I wanted to drive towards. And I think that OpenAI can still basically wants to build a great tool.”

“There is a fundamental tension or trade-off between centralizing and decentralizing, and those are really the only two moves.”

“You just sort of inevitably wind up on a 18, 24 month, 36 month beat, having a reorg at some level. And that's not something to fear, but something that you should embrace.”

“The particular object level advice would be that you don't build communities, you grow them. Your communities are more like a garden than they are a building. And like a garden, it's important that you, there's things you can do to make the garden more likely …”

“And so when we talk about organic alignment I think the important thing to recognize is that alignment is not a thing. It's not a state. It's a process.”

“Humans are more relatively goal coherent. Than any other object I know of in the universe.”

“Goals are one level of alignment. You can align something around goals. The kind of goals we're talking about here are one level of alignment. You can align something around goals by like if you can explicitly articulate in concept and in description, the stat…”

“If under, if its surface level behaviors looked like a human, and then if after I probed it continued to act like a human, and then I continued to interact with it over a long period of time, and it continued to act like a human in all ways that I understand a…”

“That's like the pause AI people. I think that's totally unrealistic and silly”

“So that's our goal. It's like big multi-agent reinforcement learning simulations, which create a surrogate model for alignment.”

“He doesn't believe it's possible to build an AI that we meaningfully can know cares about us and that we can care about meaningfully. He doesn't believe that organic alignment is possible.”

“The good AI future is that we figure out how to train AIs that have a strong model of self, a strong model of other, a strong model of we.”

“I knew when I took that job, I told them when I took that job, that like, this is like, you have me for max, 90 days.”

“To even have an AI creature that cared about the other members of its pack and the humans in its pack, the way that like a dog cares about other dogs and cares about humans, would be an incredible achievement and would be, would, even if it wasn't as smart as …”

“It's not building great, beautiful software to win. It's building good enough software and great sales.”

“Watching a movie for free on a service instead of buying the movie to watch it is a substitution for what you're doing. But watching someone else play a game is actually marketing for the game.”

“We talked to 40 VCs, everyone turned us down.”

“What's interesting about Twitch is we actually have this very thriving middle class, because big streams monetize much more poorly per viewer than medium-sized streams do. And so there's this, most of the money actually gets made Not in the top tier of streame…”

“Unlike most social media, You can't really go viral on Twitch. It's almost, because it's live, it can't go, you can't get to, oh my god, look at this, millions of people see it. It's like, however many people saw it, it's over.”

“So in the beginning, that's, I think that's absolutely not true. Like, I think that you're a six person startup. The best CEO is either an amazing salesperson or an amazing product leader. Or a brilliant engineer, and they, whatever it is that the company real…”

“The first VC round after the pivot was with Bessemer and we struggled to raise that round. We had to pitch a lot of VCs, but Bessemer is amazing, but like we probably pitched four EVCs in that round. It was a struggle. Nobody wanted to invest in Twitch at firs…”

“For Twitch, I think the next five years is all about growing the base of the kinds of content that you can live stream. And interact around and making that really easy.”

“Most of AI is focused on alignment as steering.”

“The reason humans are so good at this is we have a really excellent theory of mind. I already know what you're likely to ask me to do. I already have a good model of what your goals probably are. So if you ask me to do it, I have an easy inference problem.”

“What changes for most agents when you're doing multi-agent training is that, like, basically having lots of agents around makes your environment way more entropic. Like agents, agents are these huge generators of, like, entropy, because they're these big, comp…”

“Basically, we built Google Calendar roughly a year before Google Calendar was released.”

“I think what YC is looking for is people who can communicate clearly, but most of all, people who, when they're challenged, they neither rebel nor acquiesce. And I think that's a, that's one of the most important traits in startup founders, actually.”

“In the beginning, the investable asset is not the idea and it's not the execution and it's not the research or whatever. The investable asset is the founders. And you're much better off investing in a good team that has no idea what they're doing or there was …”

“Actually, one of the big things we discovered was, almost immediately, people liked him taking off the camera and talking to them more than they liked almost anything he could be doing. If he was doing something really interesting, like he's taking a trapeze l…”

“We had the fire trucks called, and even if the fire trucks had showed up, we would deal with it off camera so that there wasn't a reward for the person doing it. And that it definitely over time Cut down on the amount of harassment we experienced.”

“I think Justin TV worked to some degree because our actual customer was the streamer. And because we were trying to build something that worked for us as a streamer we built something that actually worked because we deeply understood the needs of our customers…”