The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 183 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 2 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Schrittwieser: OpenAI's GDPval is a strong benchmark for economic impact
“I think that GDP is like a super cool evaluation from OpenAI where they collected a lot of, like, you know, real world tasks from real domain experts to make sure that it is actually representative of what you might do in the economy.”
Tworek: Effective reinforcement learning requires a 50/50 balance of rewards and punishments
“In a good way, the good way to do RL is if you balance those things. So if you kind of give cookies half of the time and punish the other half of the time, but this is almost like a mathematical kind of kind of aspect of it.”
Tworek: AI reasoning spent on unknown answers yields better results with time
“I think that, that, that difference is here, like answering a question usually means you already know the answer and you just elicit the answer, you know, and the process of reasoning is Getting to the answer that you don't know, and usually the longer you spe…”
Tworek: OpenAI's research division consists of slightly under 600 people
“The truth is in research at OpenAI, which is like around, like slightly less than 600 people at the moment, everyone knows everything really, really it does.”
Laskin: Pre-LLM AI breakthroughs spent more effort on environment design than model training
“And a lot of the project was in these big projects was not even on training the models. It was figuring out how the agents should actually interface with the environment that you're training it in.”
Evans: OpenAI lacks visibility into user search, purchases, and social media activity
“Back to open AI, they've got a partial view on you, but like, they don't know what you've bought. They don't know what you've searched for. They don't know where you go. They don't know what Instagram you look at and what TikTok you look at and what YouTube yo…”
Nvidia released an AI model that outperformed GPT-4
“Just a couple of weeks ago, Nvidia announced that In addition to having the best chips, they just released a model that was actually better than GPT-IV.”
Socher: AI research papers predict future commercial AI capabilities
“You can actually, if you want to predict like, oh, what's the next thing of open AI? Just read all the research papers over the last couple of years. And a lot of those will come and get into products at scale with all the right engineering and so on. And then…”
Real-time AI apps require abstraction layers to handle LLM provider outages
“Yeah, I think you have to create a layer on top of these at a minimum, right in the end, whether Entropic or OpenAI, you know, they're companies, right, they sometimes have stability issues, right, if they're down, so what they're gonna do about that? Since ou…”
Moody's Found OpenAI Outperformed Competing Models in Internal Benchmarks
“ChatGPT was the one, OpenAI was the one that was performing better in a, in over Different parameters, right?”
Sapoznik: OpenAI is aggressively commoditizing NLP and machine learning capabilities
“Where we see this going is what open AI and all that we're doing is really a really fantastic and spectacularly aggressive commoditization of NLP and some other machine learning capabilities.”
Srinivas: Jasper and Copy.ai initially generated more revenue than OpenAI
“News started spreading that there were companies like Jasper and copy.ai that started making more money than even OpenAI at the time.”
Kambadur: Triggering unsafe OpenAI outputs requires clever adversarial prompt engineering
“I've seen really huge advances from open AI's models in their responsible AI and their safety. Like now it takes a lot of clever prompt engineering is what we call, you know, using different ways of talking to the model to get it to reveal interesting stuff. I…”
Gourley: OpenAI's GPT-2 was not fact-aware
“Some of the colleagues out of OpenAI released GPT-II. That was language generation, but it wasn't fact-aware.”
Katti: OpenAI sources compute from all major cloud hyperscalers
“We have compute from, effectively, many sources. So, Microsoft, obviously, is a big partner, important partner. We also have compute, as we have announced, from AWS, Ion Google. So, we have compute from all of the hyperscalers, effectively.”
OpenAI studied reinforcement learning reasoning models internally long before releasing o1
“So before we released a one and thinking reasoning models, we were studying this internally”
OpenAI's RL team researches models two generations ahead of current releases
“And to do that, we need to make thinking models and some, somewhere along the way, we interact with that process. Usually at the earlier stage for models, you know, not the next model, but things that are like the next model or the next, next model.”
OpenAI focuses on reducing AI agent error probability per time interval
“What we've been pushing a lot on is, like, making sure that the model, like, we decrease this probability of being wrong every, like, two minutes.”
Dubois: A large portion of OpenAI post-training team works on model-as-a-judge frameworks
“A lot of my team works on that, and I think it's really critical is to work on this model, model as a judge kind of framework.”
Kolter tried recruiting Schulman and Karpathy to CMU before OpenAI launched
“I was trying to get both John Schulman and Andre Karpathy to apply for faculty jobs at CMU, and I was trying to understand where they were, if they were going to apply, what they're going to be like, and they said, no, I think I'm going to be doing this startu…”
Izmailov: OpenAI had three alignment and safety teams during his tenure
“Even at OpenAI, when I was there were three teams related to alignment and safety.”
Kaiser: Distillation lets OpenAI combine research projects without long pre-training runs
“With distillation, you have the ability to put a number of projects into one model. It's kind of nice that you don't need to wait on all of them to complete at the same time. You can try to periodically put this together, actually make sure that as a product i…”
Kaiser: OpenAI's frontier AI models do not yet have 100 trillion parameters
“Our models don't have a hundred trillion parameters yet.”
Kaiser: Pre-training consumes the most GPUs of any AI development stage
“Currently, pre-training just uses the most GPUs of all the parts, so it needs the most GPUs, right?”
Tworek: OpenAI's competitive programming performance was a research byproduct
“I think we used like specifically programming puzzles for a while as a, Very nice research test bed of our ideas. Those are nice problems to experiment on them, and they weren't ever, like, considered part of the product, but it's, those are pretty, like, comp…”
Tworek: OpenAI internal teams heavily use Codex for coding
“We definitely use codecs a lot for coding, and this is only getting, getting better.”
Tworek: OpenAI placed second in AtCoder behind a former employee
“We did also like IOI, International Olympic Informatics earlier this year, and Adcoder Heuristics competition as well, where we went second, we, behind a single human that is also a Polish person that used to be employed by OpenAI some time ago.”
Humphreys: Canva built AI model abstraction layer over OpenAI, Bedrock, and SageMaker
“We've invested enormously in putting a platform abstraction over a lot of the third party AI that we use whether that's kind of like base layer infrastructure like Bedrock or SageMaker, or whether it's third party AI providers like OpenAI, and that allows us t…”
Levie: Box officially supports Gemini, Anthropic, and OpenAI models
“We officially support Gemini family. We support anthropic, you know, variety of cloud models and then support open AI and the GPTs.”
Lin Qiao: OpenAI switched completely from TensorFlow to PyTorch
“OpenAI switched to use PyTorch fully.”
GPT-4 price per token dropped roughly 90% in one year
“The, I think the price per token of GPT-IV dropped something like 90%. Over the last year”
ZoomInfo selected Anthropic over OpenAI and Google for call summarization
“We are using Anthropic, we are using OpenAI, we are using Google, but for that one, we released Anthropic.”
OpenAI inference pricing drops 160x between model generations
“If you look at the pricing page for, like, OpenAI, and you look at the 4.5 turbo, and the cost per inference, and compare it to the next most recent model, there's a 160 X difference in pricing.”
Traynor: GPT-4 crossed the hallucination threshold required for customer support bots
“So Finn's built on GPT-IV, by the way, we've, we tried, we wanted to build on a three, 3.5, but it didn't, it's still, I remember back when we used to talk about hallucinations, like four was the sort of the perceptual change for us in terms of trust and relia…”
Van Luijt: OpenAI recommended Weaviate when deprecating its search endpoint
“OpenAI had a they had a search endpoint that they deprecated. This is pre-ChatGBT, so, but they, so they deprecated that search endpoint, And say, well, you know, you just can buy our embeddings. There are three ways that you can index them and search for them…”
Glean AI shifted 100% of its machine learning stack to LLMs
“Over the course of the last 12 to 18 months, we've done a hundred percent shift to LLM modeling for the complete stack. And we're using Vertex and OpenAI.”
Polu: OpenAI combined research and engineering early with fluid career ladders
“I think I saw the transition from research to research plus engineering being the ingredients into building the largest and best models. But even very early on OpenAI was very open on the fact that you could be one or the other Quite fluidly and changed betwee…”
Sternberg: Notion uses models from both OpenAI and Anthropic
“While those are the two that I can say are, yes, publicly, like, we have leveraged both of those partners”
Kambadur: OpenAI sought top 10th percentile doctors and lawyers as data annotators
“Their model itself has gotten so good that they now need, like, extreme experts to be able to train the model, so we saw an ad for their annotators, which are the people who, like, help give feedback to the model, and they asked for, like, top 10 percentile ch…”