The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 15 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Roucher: DeepSeek-R1 ranks slightly below OpenAI o1 on smolagents tasks
“I tried R one, but R one is a bit under O one with small agents. And I think this is also a matter of formatting. Like sometimes the model struggles to just output them, the code snippets in the correct way that we expect.”
OpenAI o1 struggles with multi-step agentic tasks compared to GPT-4
“O-One is very good at programming, but it's kind of, the agent part was the harder part to get it to do here.
I think it's like less trained
To take the next step in like an agentic task, whereas GPT-IV for like the last two years has been really, you know, pr…”
Pullen: CoScene's Genie outperforms OpenAI o1 out of the box on SWE-bench
“So it was obviously great to see, like, we still are better than O-one out of the box. You know, even with an older model, and I'm sure that that, that Delta will continue to grow once we're able to train O-one and once we've done more work on our dataset usin…”
Shawn Lewis: o1 agent achieves 57% single-pass, 64% with parallel rollouts
“It solves, like, something like 57% of problems with a single
Rollout and then using parallel rollouts and selecting the best one.
With other techniques, we get something like 64%.”
Hylak: Do not micromanage OpenAI o1's step-by-step reasoning when prompting
“What I found that worked really well was not telling the model how to think about it. If that makes any sense, like almost more like you're actually interfacing with a person, which is like scary in its own right. But no, if you're working with a like a cowork…”
Swyx: Claude Sonnet and Gemini Outperform o1-Preview in Coding
“Claude Sonnet so far is beating O-one on coding tasks without At least one preview without being a reasoning model and same for Gemini pro or Gemini two point O.”
First-principles GPU kernel modeling was beyond AI capabilities before OpenAI o1
“O1's like the initial thinking models were a big deal when I was doing like core academic, like how do I create a performance model for explaining how this kernel behaves? Like, From first principles, that kind of thing was not really in the scope of any model…”
Lambert: Reasoning models solved basic skills; planning is the next frontier
“So I came up with four and the foundational one was skills, which is What I would say that we have already done with O-one and R-one, which is you do a lot of RL, you show the inference time scaling works and you get really high benchmark numbers. And then the…”
Cosine's fine-tuned GPT-4o scored higher on SWE-bench than OpenAI's original o1
“And then we also on this podcast, we interviewed Cosign that actually fine tune four O to, on three bench, sorry to achieve data on three bench. And that score was actually higher than O one when it came out.”
Hylak: AI development is returning to multi-step chaining of specialized models
“It feels like we're, like, going back Into a time where having these separate steps with almost like varying levels of intelligence becomes increasingly important, like this idea of chaining.”
Fanelli: Chat interfaces confuse o1 users, who should write specification briefs instead
“It's almost like the chat is still the UX, you know, and I think that's what confuses people. Maybe like if you didn't have a chat interface and it was like write a brief, you know, I think people would maybe be steered more the right way.”
Pullen: Replacing Genie's reasoning traces with o1 traces improves performance
“Even now we've started Replacing some of the reasoning traces in our Genie model with reasoning traces generated by O-one, or at least in tandem with O-one, and we've already started seeing improvements in performance from that point.”
Weil: OpenAI o1 will support function calling by end of 2024
“I'm really excited to see things like system prompts,
And structured outputs, and function calling, make it into a one, we will be there by the end of the year.”
Hylak: Non-engineers form fixed negative mental models after one failed AI test
“Most people try something once and if it doesn't work, they like They have this like very fixed mental model. They never tried again.”
McAteer: Use coding assistants to bootstrap structured prompts from raw brainstorms
“What I do, and I think what you should do is use other prompts to sort of bootstrap your prompts. So like I have it set up in cursor or in windsurf where I was telling Alessio before we started, you basically, I have like a prompt template in a prompts directo…”