The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 151 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 4 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
McKinzie: OpenAI is developing models with precise uncertainty understanding
“And I hope we can get to a place where our models have a more precise understanding of their own level of uncertainty. Because you know, if they already know the answer, they should just kind of tell you it. And if it takes them a day to actually figure it out…”
Mitchell: Physical time bottlenecks make AI tasks harder than simulatable domains
“Stuff that is really bottlenecked by like time, like the physical world is also, you know, just harder than stuff that we can simulate really well.”
McKinzie: Multi-agent RL is a good baseline for human collaboration
“There's no reason you can't scale all this up so that models are trained to be really good at cooperating with each other. I mean, there's a lot of already existing literature on multi-agent RL and yeah, if you want the model to be good at something like colla…”
Mitchell: OpenAI plans to unify models and remove the ChatGPT switcher
“You know, I think for us, like unification of our models is something that, you know, Sam has talked about publicly that, you know, we have this big crazy model switcher in ChatGPT and there are a lot of choices and you know, we have a model that might be good…”
McKinzie: OpenAI models use external tools with weirdly human-like intuition
“It's also surprising to me how intuitively our models do use the tools we give them access to. It's like weirdly human-like, but I guess that's not too surprising given the data they've seen before,”
McKinzie: OpenAI models hit an inflection point navigating internal codebases
“I think our models are getting a lot better very quickly at being actually useful. And it seems like they were kind of reaching some kind of inflection point where They are useful enough to want to reach out to and use like multiple times a day for me at least…”
Mitchell: o3 output distribution makes single-prompt evaluations misleading
“O-three can do really cool things, like when it chains together a lot of tool calls, and then, like, sometimes for the same prompt, it won't have that, you know, moment of magic, or it will, you know, just take a little, it'll do a little less work for you, an…”
Mitchell: o3 autonomously executes multi-step tasks using integrated tools
“Not only is the model it's on its own smarter than our previous O series models, which is great, but it's also able to use all these tools that like further enhance its abilities and whether that's doing like research on something where you want up-to-date inf…”
Mitchell: OpenAI limits model agency due to asymmetric error costs
“There's a reason we don't go hog wild and say, like, oh yes, here's, like, the keys to the kingdom, like, have at it. There are still, you know, asymmetric costs to, like, the time you can save and the types of errors you can make, and so we're trying to, like…”
Mitchell: High-quality evaluation benchmarks are underappreciated compared to training data
“I mean, yeah, like you want, you know, good data to train on and that's of course valuable for making the model better, but I think it is often neglected how also important it is to have high quality data, which is like a different definition of high quality w…”
Fulford: OpenAI Prioritized Read-Only Synthesis Over Action Agents
“Yeah, so I think before we focused on taking right actions, which those are examples of taking right actions, we wanted to get really good at synthesizing information from a large number of sources and mostly read-only tasks.”
RL models only need task and outcome definitions to learn research trajectories
“The cool thing with RL is that you don't necessarily need to
Know the whole process of how the person would do the research.
You just have to know what the task is and what the outcome should be, and the model will just learn during training how to get from th…”
Fulford: OpenAI Built Deep Research by Fine-Tuning o3
“Yeah, I think also the base model, or the model that we started fine tuning from O three is just a very capable model. It's trained on many different data sets, including a lot of coding and reasoning and math tasks.”
Fulford: OpenAI Trained Deep Research to Always Use Maximum Thinking Time
“I think we made a decision when training the model that We just are going to go for max thinking time every time.”
Fulford: Deep Research next steps include accessing internal documentation and GitHub
“As to deep research, I think obvious next steps for deep research would also be to have access to private data, like be able to do research over, you know, any internal documentation or GitHub, whatever it is.”
Fulford: Training reasoning models on math and coding generalizes to writing
“So I think in general you will always get a model better, better at a specific task if you train on that task, but we also see a lot of generalization from training on one kind of task to, you know, other domains. So you can train a reasoning model on mostly m…”
OpenAI to release mid-tier features bridging quick search and Deep Research
“And I think that we will release things soon that people will be happy about and we'll fill that gap.”
Guo: Sam Altman stated OpenAI will support Model Context Protocol
“And Sam from OpenAI said, like, they're going to support it as well”
Guo: Harvey has raised over $500M from OpenAI, Sequoia, Kleiner, and others
“They've now raised more than five hundred million dollars from investors such as OpenAI, Sequoia, Kleiner Perkins, GV, Bloodgill, and me.”
Guo: Deep Research raises the bar for entry-level knowledge work
“The biggest deal to me is that it immediately raises the bar for a number of different types of knowledge work where like a, you know, where I might have hired a median intern or analyst before. I mean, we don't do that here, but like where one could hire a me…”
Sarah Guo says test-time compute scaling unlocks new AI competition
“Another school of thought is which I do subscribe to, by the way, is you know, new scaling law, right? So will allow us to do an important range of new tasks, and how good it is exactly at this moment is not the important thing. It's a new dimension of competi…”
Goyal: Some enterprises consolidated AI stacks to OpenAI, AWS, and Braintrust
“There's some companies that we talked to and their AI vendors are, it's literally OpenAI, AWS, and Braintrust and pretty much everything else has consolidated away.”
Alexandr Wang: Scale partnered with OpenAI on the first GPT-2 RLHF experiments.
“So we partnered with OpenAI at that time to do the very first experiments on RLHF on top of GPT-II.”
Gil: Multiple AI Labs Have Held Back Voice Cloning Tech for Years
“My sense of the market is that multiple players have this technology, but they've been holding it back. And in some cases they may have had it for a year or two now.”
Gil: Mistral reached near GPT-4 capability within one year of founding
“They went from basically starting the company to almost GPT-IV level in less than a year.”
Hoffman: Pre-release GPT-4 gave useless MBA-style advice on AI investing
“When I first got access to GBD-IV you know, months before it was publicly accessed, because I was on the OpenAI board, I sat down and said how can I, Reid Hoffman, make money through investing in artificial intelligence? Because I just wanted to try it. And it…”
Gil: Market will always support alternative LLM vendors for leverage
“In other words, you know, one could argue that no matter what OpenAI does, there'll always be at least one or two other suppliers or vendors or partners
For advanced LLM simply because the market always wants an alternative, even just for negotiation leverage.”
Guo: Founders Will Avoid Mixed-Incentive Board Structures Post-OpenAI
“I think a lot of entrepreneurs are likely to think twice about placing their destiny in the hands of groups with explicitly mixed incentives now.”
Halligan: HubSpot tried to invest in OpenAI alongside Microsoft but got rejected
“The second we saw ChatGPT, we tried to invest in, in OpenAI. Dharmesh had got early looks at it, and we tried to get in the round with Microsoft, and had a bunch of discussions with them, and they bought, at the end of the day, they wouldn't let us in, which i…”
Rauch: OpenAI is by far the most popular backend in Vercel telemetry
“OpenAI is by far the most popular backend. Like we have sort of aggregated anonymized telemetry and like, what are the backends that our server functions are talking to and OpenAI is sort of Biggest.”
Khan: Bill Gates challenged OpenAI to build an AI passing AP Biology
“It turns out that Bill Gates, when he saw GPT-III, he's like, oh, this is cool. But we all know GPT-III really didn't have a good handle on knowledge, and he told the OpenAI team, he's like, I'll be impressed if this could pass the AP Biology exam. He literall…”
Khan: OpenAI approached Khan Academy before finishing GPT-4's first training run
“They hadn't even finished the first training run of GPT-IV, but they said, you know, we think it's going to be done in about two weeks, and we think This is going to be the model that really wakes up people to the power of generative AI. We want two reasons wh…”
Khan: OpenAI does not use Khanmigo student interactions for model training
“None of the interactions, and this is to credit to open AI, none of the interactions are being used to train the AI.”
Pomel: No Competitor Has Caught Up to OpenAI's Frontier Model Performance
“Nobody has quite caught up to open AI yet in terms of what the frontier model is and the maximum level of performance you can get.”
Kevin Scott: OpenAI Was the Highest-Ambition AI Partner Available
“And when we looked around, like OpenAI was clearly the highest ambition partner that was in the field, you know, and I think still their ambition is just breathtaking in what it is that they're trying to accomplish.”
Kevin Scott: OpenAI's size and energy resemble early Google
“When I go sit In OpenAI, it really reminds me of early Google days, and it's about the same size Google was when I joined, and so, like, I couldn't figure it out for a while, and I was like, wow, this is, like, really giving me, like, you know, early Google no…”
Gil: OpenAI Is Hiring String Theorists into CS Roles Like Early Google
“It's the first time I've seen, like, string theorists getting hired again. Yeah. Into computer science roles. Yeah. You know, since Google days.”
Guo: No open-source model matches GPT-3.5, GPT-4, or Claude quality
“So there's nothing out there today in open source that is like GPT four, three, five or anthropic cloud quality, right? So there is a, there's one player out in front and that's open AI”
Graveley: GitHub Copilot's in-the-wild test pass rate rose from under 10% to over 60%
“And eventually, you know, it went from, you know, less than 10% on in the wild test to over 60%. So that's like one in two tests it can just generate code for.”
Ramaswamy: Action Transformers Will Be Powerful but Remain Extremely Nascent
“I think, like, action transformers is going to be an incredibly powerful area. The technology is very nascent, so unlike, say, you know, OpenAI's ability to crank out new generations of LLMs, I don't think that tech is yet at a point where people can build lot…”
Gil: Five years ago, Anthropic barely existed and OpenAI was early
“Anthropic basically didn't exist five years ago. OpenAI was still quite early. I think GPT-III just came out and SpaceX was trading at 80, a hundred, something like that.”
Nadella: Microsoft-OpenAI partnership began around scaling compute on transformers
“Obviously I got into all this when I got excited by the scaling laws paper and, you know, when, You know, even the OpenAI partnership came about when those folks said, hey, we're going to really throw a lot of computer transformers and they've helped, right?”
Krishnan: DeepSeek Preceded Anthropic and Google in Releasing Reasoning Models
“DeepSeq was the only reasoning model, which was not open AI. So I don't think Claude had come out of the reasoning model yet. I don't think Google had yet. It was the only non-open AI reasoning model.”
Mitchell: AI offloads tasks lacking comparative advantage to external tools
“I think like part of this is you can just allocate compute a lot more efficiently because you can defer stuff that the model doesn't have comparative advantage to doing to a tool that is like really well suited to doing that thing.”
Fulford: Pre-existing training datasets exist for math and coding, not browsing
“With the math and coding problems that people were already training on, those data sets already exist. You know, you can have a math problem with a ground truth answer, and you can train on those. But for browsing, it's kind of more open-ended. You don't reall…”
Fulford: Deep Research Uses a Text Browser and Python Execution Tool
“So right now we just have the browsing tool, which is a text based browser, but it can see embedded images and like open PDFs. And then also it has access to a Python tool so it can do analysis and calculations and plot graphs and things like that.”
Fulford: Deep Research Began as a Prompted Demo Without Model Training
“We initially had built a demo to pitch people on this idea and it was no model training involved. It was fully just prompted models with the UI pitching the vision of what this product could look like.”
Torres consulted for OpenAI on LLM apps for NBA and Ticketmaster
“And so I did consulting for them and I started from the bottom, right? People were like, who are you? I was doing some support tickets for them et cetera, explanations. But then I started getting contracts like the NBA, you know, ticket master. I'm in Dubai ta…”