Feb 27, 2025 · 1h 47m · lennys-podcast
An inside look at X’s Community Notes | Keith Coleman & Jay Baxter
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Keith Coleman and Jay Baxter explore the technical architecture, mathematical bridging algorithms, and lean operating philosophy behind X's Community Notes. They explain how decentralized, open-source crowdsourcing successfully combats online misinformation, reveals societal consensus across ideological divides, and transforms platform governance.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 24.2% of the talking time here. How this is scored →
Lenny as informed peer 2.9 Guest teaching 5.3 Guest disagreement 0.3 Lenny pushing back 0.6
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:46 Jay rejects the 'fact-checking' label
Jay politely but firmly interrupts to correct Lenny's characterization, insisting Community Notes is strictly about adding missing context rather than arbitrating truth like traditional fact-checkers.
Hardest push from Lenny ▶ 17:05 Lenny offers direct user experience pushbackLenny challenges the current notification UX, pushing Keith on the need for push notifications to explicitly summarize what was misleading rather than forcing users to click through.
Biggest teaching moment ▶ 18:37 Jay and Keith correct Lenny's threshold mathJay and Keith correct Lenny's assumption that a 0.4 matrix factorization score equates to a 40% agreement rate, explaining the mathematical mechanics behind the scoring threshold.
Lenny holds their own ▶ 52:58 Lenny synthesizes the high-velocity operating modelLenny synthesizes the fragmented operational insights into a structured, executive-ready framework for building high-velocity, autonomous product teams.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Preview Highlights: The Power and Architecture of Community Notes | 0 | 0 | 0 | 0 | Introductory monologue featuring teaser soundbites and sponsor reads for WorkOS and ProductBoard. Lenny establishes the high-impact framing of Community Notes. | |
| Understanding the Fundamentals and Bridging Algorithm of Community Notes | 2 | 6 | 1 | 1 | Lenny asks how the algorithm functions, and Jay explains the bridging-based agreement model which searches for consensus among historical disagree-ers rather than taking simple majority votes. | |
| Everyday Applications, Contributor Profiles, and Context vs. Fact-Checking | 3 | 6 | 2 | 1 | Jay corrects Lenny's terminology, emphasizing that Community Notes provides context rather than traditional top-down fact-checking. Keith adds details about random contributor selection. | |
| Universal Eligibility Rules and Earning Note-Writing Privileges | 3 | 6 | 0 | 1 | Keith and Jay outline how users earn note-writing privileges and explain media matching to scale notes across duplicated images and videos. | |
| Closing the Misinformation Latency Gap via Retrospective Notifications | 4 | 7 | 2 | 2 | Lenny offers direct user feedback on push notifications, and later asks if a 0.4 score means 40% agreement. Jay and Keith educate Lenny on matrix factorization scores and how quality thresholds operate. | |
| Handling Extreme Polarization and Harnessing the Entirety of Humanity | 2 | 6 | 1 | 1 | Jay and Keith explain how the algorithm models hyper-polarized topics and outline why having diverse viewpoints from all of humanity is crucial for accurate signal extraction. | |
| Volunteer Motivation and Holding Major Institutions Accountable | 3 | 6 | 0 | 0 | Keith recounts the White House modifying an official statement due to a note written by a regular user, and Jay shares quantitative A/B test data showing 50-60% drops in reshares. | |
| Origin Story: Frustrations with Centralized Trust & Safety at Twitter | 3 | 5 | 0 | 1 | Keith details his frustration with managing large PM teams and traditional centralized Trust & Safety models at Twitter, prompting his pivot to building Community Notes. | |
| Naming History: From Birdwatch to Community Notes Under Elon Musk | 4 | 4 | 0 | 1 | Keith recounts the transition from Birdwatch to Community Notes after Elon Musk acquired the company and messaged him directly about the product. | |
| The 'Thermal' Operating Model for Autonomous, High-Velocity Teams | 4 | 5 | 0 | 1 | Keith explains the 'Thermal' internal startup model created under Kayvon Beykpour, which granted absolute team autonomy and direct reporting lines to leadership. | |
| Early Team Composition and Initial Anti-Manipulation Graph Models | 3 | 6 | 0 | 1 | Jay explains why initial PageRank-style graph algorithms failed against ideological bias, leading the team to run an internal ML competition to discover the bridging agreement model. | |
| Lightweight Team Operations: Daily Syncs and Single Living Documents | 5 | 5 | 0 | 1 | Keith and Jay describe running the team using a single massive Google Doc without formal task trackers, and Lenny synthesizes the organizational takeaways. | |
| Internal Recruiting and the Decisive Power of Employee Self-Selection | 3 | 5 | 0 | 0 | Jay and Keith discuss internal recruiting, career risk, and the necessity of self-selection, drawing parallels to Musk's 'fork in the road' email. | |
| Operating Under Elon Musk and Shipping Infrastructure at Radical Pace | 3 | 5 | 0 | 1 | Keith shares the example of scaling Spaces for massive concurrent audiences in three weeks to illustrate the velocity advantages of lean, unencumbered teams. | |
| Lean Operations Post-Acquisition: Code Deletion over Accumulation | 3 | 6 | 1 | 1 | Jay explains that deleting code is often more valuable than writing new features to minimize maintenance burdens, slightly refining Lenny's assumption that systems had to be rewritten from scratch. | |
| Early Piloting, Setting Low Expectations, and Elon's First Reaction in 2020 | 3 | 5 | 0 | 0 | Keith tells the story of early public piloting, considering a dumpster-fire GIF to lower expectations, and Elon Musk's unsolicited 2020 tweet praising the prototype. | |
| Foundational Principles: Open Source Transparency and Public Ownership | 3 | 6 | 0 | 1 | Keith explains the non-negotiable principles: no manual administrative overrides on notes, open data, public code, and universal contributor access. | |
| Engineering Challenges of Open-Source Reproducibility and Auditability | 2 | 6 | 0 | 1 | Jay describes the engineering trade-offs required to make large-scale matrix factorization executable on a single machine for external open-source auditors. | |
| Crisis Stress-Testing: The Israel-Hamas Information Deluge | 3 | 5 | 0 | 0 | Keith describes the information explosion during the October 2023 Israel-Hamas conflict and how recent speed optimizations reduced note turnaround times to five hours. | |
| Refuting the Polarization Myth: Uncovering Broad Consensus in Society | 3 | 5 | 0 | 0 | Jay and Keith discuss how Community Notes disproves the belief that consensus is impossible in polarized environments, and Lenny highlights how it encourages healthy media skepticism. | |
| The Counter-Intuitive Power of Contributor Pseudonymity | 3 | 6 | 0 | 0 | Keith reveals why pseudonymity increased contributor honesty and willingness to cross partisan boundaries, running counter to the initial assumption that verified identities build trust. | |
| Organizational Resilience Across Five Leadership Transitions | 3 | 5 | 0 | 0 | Keith explains how rigorous incremental data proof and low personal ego enabled the project to survive five leadership regimes without being derailed by politics or cost-cutting. | |
| The Future of Community Notes: AI Collaboration and 'Super Notes' | 2 | 6 | 0 | 0 | Jay and Keith detail future roadmaps, including 'Super Notes'—an open research collaboration using LLMs to synthesize candidate notes and simulate jury evaluations. | |
| Emergent Community Use Cases: Sports Rivalries and Anti-Spam Moderation | 2 | 5 | 0 | 0 | Jay notes unexpected emergent community behaviors like debating soccer rivalries (Messi vs. Ronaldo) and anti-spam moderation. Keith shares closing optimistic reflections on scaling bridging algorithms. |
Statements from this episode (52)
Baxter: Bridging agreement between polarized users ensures accuracy and neutrality
“I think the key thing really that we do is we actually look for agreement from people who have disagreed in the past. And what we see is when people actually have that sort of surprising agreement, that's what makes the notes so neutral and accurate and well w…”
Baxter: Bridging algorithms enable crowdsourced moderation without external fact-checker labels
“If you said I think like back in 2020, before we started building anything here, whether this could work at all, I think a room of ML engineers would say, oh, you have to keep it closed source. You know, people are going to be manipulating this all the time. Y…”
Coleman: Community Notes rejected using journalists to avoid curated editorial decisions
“Initially, some people inside the company were suggesting like, Hey, why don't you have journalists or, you know, some select group be the first participants, but Very specifically, we're like, no, that's like, we're trying to move away from the idea of curate…”
Baxter: Community Notes provides missing context rather than traditional fact-checking
“One distinction that I would make, which may maybe can come off as nitpicky, but I think is important is community notes adds additional context. It's not fact checking necessarily. So So there are cases where the post could be true, but maybe it's just mislea…”
Coleman: Every post on X is eligible for Community Notes
“So every post is eligible for notes. It's, and that was again, another really important principle. It's like, It, we shouldn't exempt Elon. We shouldn't exempt government figures. We should, like everyone, even advertisers can get notes. So any posts on the pl…”
Coleman: Users must earn Community Notes writing privileges by rating
“For a note to be proposed, the person proposing has to have earned the ability to write notes. So there is that, that aspect where you have to, like, earn in to be able to do this. And the way you earn that ability is, is through your ratings by demonstrating …”
Coleman: Community Notes generated 30B views across 95k notes in 2024
“And over, ah, let's say the last year, 20, 24, we had something like 95,000 notes that were seen about thirty billion times. That's more than double the prior year. Prior year was something like 37 K notes seen fourteen billion times.”
Coleman: Community Notes has roughly 950,000 global contributors
“There's something like 950,000 contributors around the world. That's, you know, nearing a million people making this happen.”
Baxter: Zero-click, specific in-feed context is essential for fact-checking
“So, so pulling the context up, you know, so that you have zero clicks that you need to make and keeping it specific is so important.”
Coleman: X notifies engaged users when Community Notes appear on posts
“Inevitably there's going to be a time gap between when a post goes live and when people figure out what's going on and when they get the note out there. And so we send those notifications to try to close that gap.”
Coleman: Social media corrections reach users far more effectively than print
“It's also a pretty cool example of something you can do on the internet in the social media world that was difficult in kind of like a print or standard news world where you would see maybe a correction, like the next day in a corner of a paper, but it was har…”
Coleman: Community Notes only displays 7% to 11% of proposed notes
“We probably show about eight percent of notes that get proposed. I think that's, it's been between, let's say, seven percent and 10% or 11%, something like that over time.”
Coleman: Authors of unhelpful Community Notes lose their writing privileges
“If you write one that people, you know, people who normally disagree, find not helpful, you actually will ultimately lose your ability to write and have to earn it back.”
Baxter: Independent studies show Community Notes change user belief in claims
“Actually, there have been external studies, you know, run by people totally independent of us who have found that if you take a community note or a post with or without a community note, that actually people's agreement with the core claims in the post does ch…”
Baxter: Community Notes algorithm stops counting ratings from consensus-violating users
“I would also say there is this reputation component to the algorithm as well. So if you consistently rate notes in a way that is counter to the bridging based consensus, then we'll stop counting your ratings, right? So you know, if you're the kind of person wh…”
Coleman: Extreme viewpoints are necessary data for Community Notes to find broad consensus
“Our view is it's actually, we want to have all of humanity here, because if we have all of humanity, we can, we then have the data to understand what notes will be helpful to actual humanity. You know, we can better model that better or better understand and b…”
Coleman: Community Notes prompted the White House to delete and reissue a tweet
“When we first launched US wide, this was like in two, a note appeared on a white house tweet. And the White House deleted the tweet and reissued an updated statement.”
Baxter: X core ranking algorithm does not demote noted posts
“Although we don't actually use the fact that a post was noted in the core ranking algorithm”
Baxter: Notes drop likes and reposts by 30% to 40% in internal A/B tests
“We saw more like 30 to 40% engagement rate drops for likes and reposts and an AB test we ran. When comparing showing a post with or without a note”
Baxter: External studies show Community Notes reduce total reposts by 50% to 60%
“Multiple different external research groups have both found consistently that There's like a 50 or 60% drop in total reposts, which is just nuts after a node is applied.”
Baxter: Authors are 80% more likely to delete posts after receiving a Community Note
“Authors become 80% more likely to decrease, or sorry, to delete their post after they get noted.”
Coleman: Centralized trust and safety cannot scale or maintain user trust
“Nothing could handle the scale of the problem. Nothing could handle the speed. And a lot of people just didn't trust the existing approaches. Like this, the existing approaches were either fact checkers or internal trust and safety teams making decisions about…”
Coleman: About one-third of Twitter employees were leaving annually in 2016
“Something like a third of employees were leaving every year. So just imagine, like, a third of your team gone every year.”
Coleman: Dorsey mocked Community Notes as a Facebook name while Meta copies it
“Jack has made fun of it, calling it like the ultimate Facebook name or something like that, but the most boring Facebook name, which is funny because they're now, you know, launching community notes.”
Coleman: Community Notes escalates decisions directly to Elon Musk
“If we need, need something or have a question on something, I talked to Elon and it was, you know, it was like that from the beginning. It's like that now.”
Baxter: Corporate OKR planning took longer than completing entire goals
“The whole OKR determination and planning process took longer than it would take us to pick a goal and then execute it on it and finish it.”
Baxter: Community Notes' initial PageRank algorithm amplified majority bias
“The actual first algorithm that I put into production was very focused on anti manipulation. It was this kind of page rank variant but it didn't solve the problem of, you know, bias basically. So if there are some, if there are more users on one side, the, a p…”
Coleman: Community Notes runs off a single long-running Google Doc
“There is a very long running doc that has had to be chopped and purged because it was breaking Google Docs and Chrome at various points in time. It's sort of like a note taking doc. It's really where we coordinate what we're doing.”
Baxter: Google Docs beats Asana by letting irrelevant tasks fade naturally
“My memory of it is that it spent, you spent more time in the meeting grooming a backlog of irrelevant stuff than actually, you know, talking about the proper priorities. So I think it's nice in the Google doc that if something becomes irrelevant, it can kind o…”
Coleman: No employees were assigned to Community Notes; all self-selected
“We did not assign anyone to this project. Like people reached out to join or they applied to join the job. You know, I and the team interviewed every single person that joined the team and we're like, we want that person on the team. They want to be on the tea…”
Coleman: Musk's 'fork in the road' email was a great downsizing approach
“I thought that was a great approach to taking a large company and getting it down to people who are really excited about, you know, working together on, on a mission.”
Coleman: X team scaled Spaces in two to three weeks under Musk
“When, shortly after the acquisition you know, we had this product called Spaces. It was, it had been in the product before, but it was pretty small scale. And Elon wanted to run these large spaces... This is the kind of thing at one point. Oh, that would have …”
Baxter: Engineers add permanent maintenance burdens for minor A/B test wins
“Engineers have a tendency to add these little incremental wins that actually ha add. You know, more of a long-term maintenance cost than is clear because you just run a little one month, AB test. You see this, you know, significant when they don't realize the …”
Baxter: X simplified its systems by deleting code after major layoffs
“So I think we did have to do this across the company after the big way offs and you know, systems are leaner now and they can be worked on by fewer numbers of people.”
Coleman: Early mock-ups showed users across political spectrum liked Community Notes
“When we built the first mock-ups, we had just, these were just, like, pictures of, you know, depicting what community notes might look like. We showed those to people across the political spectrum. We saw, like, hey, people really like these, whether they're o…”
Coleman: Elon Musk endorsed Community Notes prototype two years before acquisition
“This is, I think, two years before any acquisition stuff happened, Elon is just a Twitter user building rockets and electric cars and other cool stuff, and stumbles on this thing that depicts the prototype that we've been testing, and he writes back, yeah, def…”
Coleman: Community Notes lacks an internal button to override or remove notes
“First of all, there is no, we don't have a button that will change the status of the note or of a note. So if a note is showing because the people have rated it, And found it helpful. It is going to show. Like, we can't change that.”
Coleman: Bad notes indicate a system problem requiring redesign, not manual fixes
“If there's a problem with a note, it's so bad you want to do something about it. It's a problem with the system. Like we need to redesign the system to be showing good notes.”
Coleman: Community Notes publishes code updates to GitHub and data daily
“Whenever we update the code, there's, or update the scoring system, there's an update in GitHub when the data is published daily, so you can download it”
Baxter: Running Community Notes locally takes 500GB RAM and roughly one day
“Oh, like, 500 gigs. It'll take like a day if you don't do anything special to speed it up.”
Baxter: Vitalik Buterin published a blog post verifying Community Notes' algorithm
“Vitalik Peteran had a blog post where he, you know, talks about his explorations, you know, making sure the algorithm really does what it says it does.”
Coleman: Community Notes published 500 notes in the first three days of Israel-Hamas conflict
“In the first, I think it was like first three days or something of that conflict. We had 500 notes covering all sorts of different you know, like out of context imagery”
Coleman: Community Notes had a 5-hour median response time during Israel-Hamas outbreak
“In the first few days of the conflict, the median time from a post going live to a note showing up was five hours, which is like crazy fast. Typical fact checking is like two to four. It's really common to see it take two to four days. These notes were showing…”
Baxter: Directly attaching notes to posts penetrates online echo chambers
“The fact that the note applies is directly attached to the post and visible by anyone who sees the post helps, you know, cross the, those filter bubbles and can kind of I think for some people it's the first time they've actually seen counter arguments to, you…”
Coleman: Pseudonymity makes online contributors more willing to cross partisan boundaries
“People are actually more willing to cross partisan boundaries when they are anonymous or pseudonymous than when they are under their real name.”
Coleman: Biggest shift from legacy moderation to Community Notes happened under X
“The biggest change there happened in X. The biggest changes prior to that were just the decision to put this out there and you know, and, Have it be operating at, you know, in public at first us wide scale. But yeah, then the big, the bigger switches came in t…”
Baxter: Research shows crowdsourced lay fact-checkers match professional fact-checkers
“There was, you know, original research you know, before, before birdwatch even started or community notes even started from external researchers showing that, you know, crowdsource fact checkers can do, you know, lay people can do about as well as Fact checker…”
Coleman: Community Notes was never started or maintained for cost savings
“You know, an interesting thing about that is no one ever asked us or brought up or seemed to care about anything related to cost savings in this process. And I think that's like a pro like an assumption people have outside the company that like, this must've b…”
Coleman: X allows users to request Community Notes and submit sources
“Recently, for example, we just released an update to what we call the community notes bat signal or the ability to request a community note. So we, anyone on X can say, hey, I think this post needs a community note, and now they can even add a source explainin…”
Coleman: Super Notes is first major external contribution to Community Notes algorithm
“Super Notes is probably the first very substantial potential change in like the algorithm of the way it works that was coming, kind of coming from the outside and plausibly could be part of the core.”