Jul 27, 2023 · 37m · how-i-built-this
When AI is your personal tutor with Sal Khan of Khan Academy
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of How I Built This Lab, host Guy Raz interviews Khan Academy founder Sal Khan on the development of Khanmigo, an AI-powered tutor built on GPT-4 designed to deliver individualized Socratic learning while safeguarding academic integrity and public trust.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Guy holds 30.1% of the talking time here. How this is scored →
Guy as informed peer 3.1 Guest teaching 2.8 Guest disagreement 0.4 Guy pushing back 0.8
speaking balance: gold is Guy, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 32:20 Sal disputes the obsolescence of writing skills
Sal firmly rejects Guy's premise that generative AI will make human writing obsolete, arguing instead that it shifts human focus to high-level editorial and investigative work.
Hardest push from Guy ▶ 22:21 Guy confronts Sal on unavoidable AI cheatingGuy presses Sal on the reality that Khanmigo's guardrails cannot prevent students from simply using third-party ChatGPT to complete their schoolwork.
Biggest teaching moment ▶ 30:05 Sal highlights lack of Algebra II in underserved schoolsSal educates Guy on systemic educational gaps, highlighting the stark statistic that many minority-majority high schools lack fundamental STEM offerings.
Guy holds their own ▶ 15:16 Guy cites Benjamin Bloom's two sigma tutoring benchmarkGuy shows deep domain mastery by citing Benjamin Bloom's foundational 1984 educational study regarding the efficacy of one-on-one tutoring.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Guy as informed peer | Guest teaching | Guest disagreement | Guy pushing back | Why |
|---|---|---|---|---|---|---|
| Khan Academy's Pandemic Surge and New Learning Initiatives | 4 | 3 | 0 | 0 | Guy references pandemic learning loss studies comparing reading and math scores. Sal contextualizes these findings with Khan Academy's internal efficacy data showing accelerated learning with dedicated practice time. | |
| OpenAI's Secret GPT-4 Demo and Early Potential | 2 | 3 | 0 | 0 | Guy prompts Sal to recount his confidential early access to GPT-4 from OpenAI. Sal explains his initial skepticism based on GPT-3, followed by his realization of its Socratic potential. | |
| Mid-Roll Break: AI Dangers and Show Promotion | 1 | 2 | 0 | 0 | After the mid-roll reset, Guy asks how Khan Academy decided to build upon GPT-4. Sal details internal organizational deliberations around math inaccuracies, hallucinations, and brand reputation. | |
| Designing Khanmigo: Socratic Tutoring, Guardrails, and Teacher Tools | 3 | 3 | 0 | 1 | Guy raises immediate classroom concerns regarding student cheating and essay generation. Sal explains Khanmigo's dual-AI supervisory architecture that alerts parents and teachers. | |
| Tackling the Two Sigma Problem in Math Tutoring | 5 | 3 | 0 | 0 | Guy displays strong research by citing Benjamin Bloom's 1984 educational study on one-on-one tutoring and the two sigma effect. Sal affirms this research and explains how Khanmigo aims to replicate human pedagogical guidance in math. | |
| Khanmigo Pilot Rollout, Computation Costs, and Feature Set | 4 | 2 | 1 | 3 | Guy pushes on downstream consequences and notes that school bans on ChatGPT won't stop kids from cheating outside school. Sal counters by highlighting Khanmigo's process-tracking feature that alerts teachers if an essay appears without drafting history. | |
| Mid-Roll Break: Exploring the Commercial Future of Khanmigo | 4 | 2 | 0 | 1 | Guy challenges Sal to put on his business hat and consider licensing Khanmigo commercially like typical ed-tech SaaS companies. Sal explains their district fee structure designed to cover underlying GPU computation costs. | |
| For-Profit vs. Non-Profit: Preserving Mission and Public Trust | 3 | 4 | 0 | 1 | Guy asks about spinning Khanmigo out as a for-profit and addresses the racial and socioeconomic achievement gap. Sal educates on equity disparities, revealing that many minority-majority schools do not offer foundational classes like Algebra II. | |
| The Evolution of Writing and Human Skills with AI | 3 | 4 | 3 | 2 | Guy suggests writing may become an obsolete human skill as LLMs improve. Sal respectfully rejects this framing, citing his conversation with journalist Kevin Roos to argue that AI elevates the requirement for editorial judgment and investigative depth. | |
| The Future of Immersive AI Role-Play and Learning | 2 | 2 | 0 | 0 | Guy analogizes AI in education to the leap from the Stone Age to the Industrial Revolution. Sal enthusiastically agrees and projects a five-year horizon for interactive, immersive historical simulations. |
Statements from this episode (16)
Khan: Khan Academy daily usage surged from 30M to 90M minutes in March 2020
“Our traffic went from about thirty million learning minutes per day to about ninety million learning minutes per day, pretty much within that week.”
Khan: 15 to 60 minutes weekly on Khan Academy prevented COVID learning loss
“What we saw in twenty-twenty-one is that the students who put in an average of 15 minutes on Khan Academy a week in a school setting, they actually saw no COVID learning loss, And that students who put in 30 to 60 minutes, they accelerated almost 40% faster th…”
Khan: GPT-4 was still pretty bad at math during early testing
“Especially in those early days, it was Still pretty bad at math.”
Khan: GPT-4 could emulate a human tutor
“This really could be that holy grail that we've all been thinking about, reading science fiction about for decades of an AI that can actually emulate a human tutor.”
Khan Academy gave 40 to 50 employees early access to GPT-4
“The first thing we did is we sat in an org-wide NDA, and I said, hey, how many folks in the organization can I get, we give access to? And eventually we got about 40 or 50 folks in our organization to get access to GPT-IV.”
Khan: Generative AI will either 10x Khan Academy or make it irrelevant
“Look, this is a transformational technology. This is one of these things where it's either going to allow us to magnify our impact by an order of magnitude, or it might make us irrelevant.”
Khan: Khanmigo Deploys a Second AI to Flag Safety Issues to Parents
“We have a second artificial intelligence that monitors conversations with the main artificial intelligence. And if any of those conversations go into a gray zone then it actively notifies the parents and the teachers that, hey, you should take a look at this c…”
Khan: AI tutoring will help Khan Academy achieve two-sigma learning gains
“Essentially, old Khan Academy or base Khan Academy trying to get to that first standard deviation. And, but now with the AI, we can go to that second standard deviation.”
Khan: Khanmigo AI evaluates math reasoning by comparing hidden expected steps
“One of the things we've done to make it more robust on the math side is when a student presents their math to the AI saying, hey, I think this is the next step, the AI behind the scenes comes up with what it thinks are reasonable responses from the student. It…”
Khan: School writing assignments will have to shift to in-class work
“And so my best guess of where, let's say, term papers or homework is going to have to evolve to is, especially in something like writing, you're going to have to do some in-class writing.”
Khan: Khan Academy costs $60M to $70M annually to operate
“It costs On the order of 60, seventy million a year to run Khan Academy, both to, you know, just our server costs are six, seven million dollars a year, and then obviously all of the engineers, et cetera”
Khan: 1M district students pay Khan Academy roughly $10 annually
“We now have about a million students in districts where they're paying on the order of about 10 dollars per student per year, To get all of these other things.”
Khan: Khanmigo costs $10 to $15 per month per user to run
“Conmigo has a very real marginal cost to it. It costs on the order of let's call it 10 to 15 dollars per month.”
Khan would consider a for-profit spin-out if foundation funding dried up
“The only reason That I could ever see trying to do some type of spin out or a wholly owned sub that's, is for some reason, if we're not getting the risk capital from the foundations, or we can't create a flywheel of sustainability, then maybe certain pieces of…”
Khan: Majority of minority-majority US schools lack Algebra II and Physics
“A majority of minority-majority schools, so that's a mouthful, but a majority of minority-majority schools don't offer courses like Algebra II, Physics, Et cetera. Not to even mention things like AP courses or IB courses.”
Khan: Immersive AI 'holodeck' simulations are 3 to 5 years away
“You know, there's a world now where you can talk to historical characters, and we're probably three to five years away where you could share a room with Benjamin Franklin or Julius Caesar or Cleopatra and immerse yourself. It'll, it'll be literally the holodec…”