Jan 4, 2024 · 1h 14m · lennys-podcast
The UX Research reckoning is here | Judd Antin (Airbnb, Meta)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this in-depth discussion with Lenny Rachitsky, veteran design and research executive Judd Antin diagnoses the systemic crisis facing user experience research, dismantles common product management misconceptions, and outlines a rigorous, multi-method blueprint for embedding business-driven research into product development.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Lenny holds 24.3% of the talking time here. How this is scored →
Lenny as informed peer 4.2 Guest teaching 5.0 Guest disagreement 3.1 Lenny pushing back 1.3
speaking balance: gold is Lenny, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:25 Calling out PM tropes as bullshit
Judd bluntly dismisses the common PM complaint that user research slows product teams down, calling it 'bullshit' and arguing that fixing preventable mistakes post-launch is what truly causes delays.
Hardest push from Lenny ▶ 59:50 Challenging the early-stage research hire pitchLenny pushes back on Judd's argument for hiring researchers in the first ten startup employees, noting that founders primarily need engineering execution and product velocity at that stage.
Biggest teaching moment ▶ 1:03:50 Deconstructing NPS flaws and demonstrating CSAT superiorityJudd delivers a masterclass on survey science, breaking down why 11-point NPS scales generate noisy data on mobile and sharing internal Airbnb findings proving CSAT is statistically superior.
Lenny holds their own ▶ 44:51 Synthesizing mental models via Patrick CollisonLenny frames the discussion with a nuanced reference to Patrick Collison's philosophy, demonstrating deep product expertise by separating user mental models from prescriptive roadmapping.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Lenny as informed peer | Guest teaching | Guest disagreement | Lenny pushing back | Why |
|---|---|---|---|---|---|---|
| Welcoming Judd Antin and Celebrating Airbnb Research Alumni | 4 | 2 | 2 | 1 | Lenny warmly opens by listing former Airbnb team members Judd managed who now lead research across major tech companies. Judd responds with humility, discussing the viral reception and critiques of his controversial article. | |
| The Macro, Middle-Range, and Micro Research Framework | 3 | 6 | 4 | 1 | Judd presents his core macro, middle-range, and micro research framework, arguing that middle-range research is a massive time sink that triggers post-hoc bias. Lenny validates the PM perspective that middle-range questions are often what PMs default to asking. | |
| The Flaws and Delays of Middle-Range Research | 5 | 5 | 3 | 3 | Lenny probes whether telling teams to deeply integrate researchers contradicts the claim that research creates bottlenecks. Judd clarifies that treating research as a late-stage service function is what causes delay, whereas end-to-end integration drives speed. | |
| The Five Essential Tools of a Modern User Researcher | 4 | 6 | 2 | 1 | Judd outlines the five critical tools of modern researchers: formative research, evaluative testing, survey design, applied statistics, and SQL/prompt engineering. Lenny accurately recaps the framework and asks how to evaluate these competencies during interviews. | |
| Bridging Empathy with Business Objectives and Revenue | 4 | 5 | 3 | 2 | Judd argues that researchers must understand shareholder reports, funnels, and OKRs to tie user empathy to business profit. When Lenny notes that PMs will cheer this take, Judd quickly counters that PMs are equally guilty of performative user-centricity. | |
| Deconstructing Performative User-Centeredness and Confirmation Bias | 4 | 7 | 5 | 1 | Judd dismantles the common PM practice of asking for late-stage validation studies, framing it as check-the-box performance rather than true learning. He emphasizes that scientific research aims to falsify hypotheses rather than confirm them. | |
| Balancing Product Intuition, Blind Spots, and Evidence | 5 | 5 | 3 | 1 | Lenny reads an excerpt from Judd's article critiquing PMs, designers, and executives for abusing middle-range research to validate personal opinions. Judd expands on how uncalibrated product intuition becomes a breeding ground for cognitive biases. | |
| Applying System 2 Thinking and Unifying Team OKRs | 5 | 5 | 2 | 1 | Lenny references Kahneman's Thinking Fast and Slow, which Judd connects to System 2 thinking and building shared OKRs. Judd defines researcher success not by separate metrics, but by becoming indispensable in decision meetings. | |
| Debunking PM Tropes: "Research Slows Us Down" and the Airbnb Button | 4 | 6 | 6 | 1 | Judd bluntly calls the trope that 'research slows us down' bullshit, explaining that quick micro-research accelerates teams. He illustrates this with the Airbnb button copy test that generated millions in revenue within 48 hours. | |
| Sponsor Break: Ahrefs SEO Tools | 0 | 0 | 0 | 0 | Lenny delivers an ad read for Ahrefs Webmaster Tools followed by a brief framing transition back into the discussion. | |
| Debunking PM Tropes: "A/B Test Everything" vs. Understanding Why | 4 | 6 | 4 | 2 | Lenny raises the common PM perspective that A/B tests offer statistical causality whereas qualitative UXR appears anecdotal. Judd explains that while A/B tests demonstrate causal effect, they fail to explain the underlying 'why' required to iterate effectively. | |
| Debunking PM Tropes: Apocryphal Quotes and Hindsight Bias | 4 | 6 | 4 | 1 | Judd debunks the apocryphal Henry Ford 'faster horses' quote and breaks down Duncan Watts's research on hindsight bias and the narrative fallacy. Lenny jokes modestly about his own career path avoiding simplistic retrofitted narratives. | |
| Refining User Mental Models vs. Dictating Product Roadmaps | 6 | 4 | 2 | 2 | Lenny cites Patrick Collison's thesis that UXR should sharpen the team's mental model rather than prescribe specific product builds. Judd agrees, warning against overreaching research recommendations that alienate engineers and designers. | |
| Breaking Research Silos and Designing Integrated Insights Functions | 4 | 5 | 2 | 1 | Judd advises companies to tear down artificial silos between UX research, market research, customer support data, and data science. He recounts consolidating these streams into a unified insights engine at Airbnb. | |
| Effective Cross-Functional Collaboration and Facebook's "Super Hiders" | 6 | 5 | 2 | 1 | Judd shares a case study from Facebook about 'super hiders' who hid newsfeed posts to achieve inbox zero. Lenny matches with his own Airbnb field trip story in Paris examining user reluctance to connect Facebook accounts. | |
| Determining Optimal Research Ratios and Early Startup Hires | 5 | 5 | 3 | 3 | Judd references Lenny's post on early startup hires, arguing that hiring a researcher early prevents costly blind spots. Lenny pushes back slightly on the reality that early-stage startups typically prioritize engineering execution over research headcount. | |
| Empowering User Researchers to Take Strategic Ownership | 4 | 5 | 3 | 1 | Judd urges researchers to stop adopting a victim mindset regarding layoffs and instead hone business communication and strategic focus. He emphasizes tailoring presentations depending on whether the audience is an IC PM or the CEO. | |
| Debunking Net Promoter Score and Advocating for CSAT | 4 | 8 | 6 | 1 | Judd methodically tears apart Net Promoter Score (NPS) as flawed survey science with arbitrary 11-point scales and poor mobile UX. He proves with empirical data from Airbnb that simple CSAT metrics provide far superior signal. | |
| The Dangers of Product Dogfooding and Misaligned Intuition | 4 | 5 | 3 | 1 | Judd warns against over-relying on internal product dogfooding, emphasizing that PMs differ drastically from end users. The conversation concludes with a rapid-fire lightning round covering leadership books, sci-fi, and stoicism. |
Statements from this episode (29)
Antin: User Researchers Must Be More Focused on Profit and Business Success
“I think researchers need to be more profit focused. And there are a lot of people out there who think that's not, I think they think that's not cool or not research's job. And I'm kind of like, well, what are we doing then if we're not helping businesses succe…”
Antin: Middle-range UX research is interesting but lacks business impact
“And then you have this middle range, which is this blobular place where the research questions are sort of middle altitude. And a lot of the core kind of, let's say user understanding questions fall here. And a lot of what research is doing is research in that…”
Antin: Middle-range UX research enables performative customer obsession
“So they yield results, which are interesting, but sometimes hard to operationalize. They trigger the post hoc bias really, really, really well, where like a lot of people can say confidently like, oh, that was kind of obvious. We knew that already. And they fu…”
Antin: UX research remains ineffective while treated as a service discipline
“As long as research is a service discipline, I think we're going to be stuck in this spot.”
Antin: Top user researchers must master five specific multi-method tools
“So the best researchers I think are, first of all, multi-method. The first iteration of user research was primarily a qualitative discipline, but a strong opinion that I have is that is largely one of those models that needs to evolve. It's not that qualitativ…”
Antin: Researchers cannot operate in A/B testing environments without basic statistics
“The best research to know a little bit of stats, like you can't interact in a world of A-B testing without knowing basic statistics.”
Antin: Average product managers perform customer centricity without real interest
“The average PM who lives in that land all day, every day. And what they do is not in the Venn. You know, I think those are people who are also like Performing customer centricity and performing user centeredness a lot when they're really not interested.”
Antin: Late-stage user research validation is performative box-checking
“Every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study, you know, just to validate our assumptions. That's user centered performance. It's too late to matter. That PM is not interested in being wrong…”
Antin: Teams should approach user research to falsify assumptions, not validate
“Like one of my big kind of mantras was we don't validate, we falsify. Right? We are looking to be wrong. That is the mindset you should use when you're approaching insights and research. I want to be wrong. I want you to do research that shows we were off base…”
Antin: Product intuition harbors blind spots that researchers must expose
“Intuition is where all of those biases lie. It's where all your blind spots are. And what great insights people do, what great researchers do when you're next to them all the time is they'll expose you.”
Antin: Product design is plagued by performative design thinking rituals
“Design, it has a ton of performance and it comes from the fact that we have like figured out user-centered design, this process or design thinking, which IDEO popularized, like that's what we're supposed to do.”
Antin: UX research teams should share identical OKRs with product managers
“I never asked for research teams to have their own separate OKRs. I said two things. Number one, what's the team's OKR? Like, shouldn't the PMs, the engineers, the designers, and the research, everybody should have the same set of metrics for success, right? B…”
Antin: True researcher success is when decision meetings require their presence
“My metric for success Is when they won't have that meeting without you. That's my metric for success. If they cannot have that decision-making meeting without the researcher there, that means you've developed influence, strong, strong trusting relationships. Y…”
Rachitsky: Teams only resist PMs when the PM is ineffective
“There's a lot of teams that are like, we don't want new PMs. We don't need product managers. They just get in the way. And I find that that's only the case when the product manager is not great and not really good at their job because most great PMs just make …”
Antin: Airbnb made millions by changing seven characters on a CTA button
“We changed the text on the button with help from our amazing content design or UX writing team. We basically changed seven characters and made Airbnb millions of dollars.”
Antin: Casual customer conversations do not constitute valid research
“Anyone can talk to a user that does not constitute research or insights work, you know, because one user can be powerful, but one user can be idiosyncratic.”
Antin: A/B tests rarely explain why user behavior changed
“AB tests are great, but one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that like flipped to Statsig. And then they're like, cool. I was significantly down over this course…”
Antin: Researchers who ask customers what they want are bad researchers
“A researcher who's going to ask customers what they want is a bad researcher.”
Antin: Best research provides recommendations without dictating exact product roadmaps
“The best research doesn't like leave it at that. Right? It tells you, and it's like the what, the so what, and then the then what. But the problem with that is some researchers go too far in the other direction where they're like, we ran the study, it yielded …”
Antin: UX researchers must learn to say no and prioritize focus
“Most researchers that I know are working on teams where they're like, I'm the only researcher. And there's like, I have seven PMs and 20 designers and like I'm trying to do 10 projects and no one's going to do a good job that way. So researchers have to learn …”
Antin: Companies must stop siloing separate research and insights disciplines
“They were like, listen, we have all these different people throwing insights over the transom. And it's great. We want to hear from the data scientists, from the product specialists, from the customer service people and the voice of the customer, whatever, all…”
Antin: User researchers cannot effectively manage more than three concurrent projects
“I used to say that a full plate for a researcher was probably three things, two big projects and a small project, right? Like a side project. More than that, your researcher is probably not doing a very good job.”
Antin: Facebook post hiding followed a power law driven by super-hiders
“The first thing I did is I looked at the distribution of hiding by user and found out that it's power law distributed, like everything on the internet. There are a few people on Facebook who hide a ton, and then most people don't hide it all.”
Antin: Germany and France are leading bellwethers for global privacy sentiment
“Germany, Germany and France were always are like bellwethers for what the rest of the world would be thinking with private data privacy concerns.”
Antin: Grow research headcount by making partner teams advocate for it
“That's how I always approached getting more headcount was not arguing abstractly for why research is important, but by asking partners who wish they had it to do the arguing for me.”
Antin: Only One B2B Company Hired a Researcher in First Ten Employees
“You published recently a list of, I think it was about 20 B to B companies and their first 10 employees. Do you remember how many researchers are in anywhere on that list? I'll give you a hint. It's between zero and two. It's one. There's one researcher on tha…”
Antin: CSAT is more precise and business-correlated than NPS
“Customer satisfaction, a simple CSAT metric is better. It has better data properties. It is more precise. It is more correlated to business outcomes.”
Antin: NPS cannot be meaningfully benchmarked across companies
“The problem with that is the research shows that NPS is idiosyncratic. So it goes up and dows in, in ways that we don't understand. And there's a lot of inconsistency in how it's asked that creates variations in the data, which means it's not apples to apples.…”
Antin: PMs must not rely on personal intuition when dogfooding products
“Doing product walkthroughs to identify lists of potential issues is a great thing to do. Prioritizing that list. Figuring out which ones are more or less a problem and for whom is an area where you should be extremely wary of relying on your own sort of opinio…”