Jul 29, 2025 · 26m · latent-space
⚡️Using RFT to Build Clinical Superintelligence
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Lightning Pod, Ambience engineering leader Brendan Fortuna discusses how advanced Reinforcement Fine-Tuning (RFT), robust evaluation pipelines, and deep clinical domain integration are transforming generative AI from simple ambient scribing into a comprehensive clinical operating platform.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 7.3% of the talking time here. How this is scored →
The hosts as informed peer 4.8 Guest teaching 5.6 Guest disagreement 0.5 The hosts pushing back 0.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 22:38 Pushing back on clinical data quality assumptions
Brendan counters Wix's assumption that acquiring private clinical data solves modeling issues, emphasizing that realistic health records are remarkably messy and uncurated.
Hardest push from the hosts ▶ 8:36 Challenging sample efficiency in RL fine-tuningWix pushes back on the unconditional benefits of sample efficiency, citing risks where models over-index or extract incorrect lessons from minimal data points.
Biggest teaching moment ▶ 20:05 Explaining why clinical data is out-of-distributionBrendan educates the hosts on how medical textbooks fail to capture the years of residency-level tribal knowledge and locked EHR documentation required for true clinical safety.
The host holds their own ▶ 20:48 Connecting residency data gaps to OpenAI Codex analogiesWix demonstrates technical domain breadth by drawing an exact parallel between medical residency data gaps and on-the-job training lessons from the OpenAI Codex team.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Comparing Autonomous Driving Machine Learning to LLMs | 4 | 5 | 0 | 1 | Wix compares autonomous driving ML workflows at Cruise with modern LLM-centric systems. Brendan explains the paradigm shift from massive labeling pipelines to sample-efficient fine-tuning while emphasizing that data engine fundamentals remain identical. | |
| Understanding Reinforcement Fine-Tuning in Clinical Medicine | 3 | 7 | 0 | 0 | Alessio asks how Ambience operationalized reinforcement fine-tuning (RFT). Brendan delivers a thorough technical breakdown of programmable graders, candidate trajectory generation, and optimizing for true downstream objectives rather than proxy loss. | |
| Preventing Reward Hacking in Structured Physical Exam Generation | 5 | 6 | 1 | 2 | Wix cautions against sample efficiency overfitting, prompting Brendan to detail real-world reward hacking incidents during physical exam generation. Brendan explains how the model inflated finding counts to boost precision and degraded clinical tone. | |
| Benchmarking ICD-10 Medical Coding with RFT | 4 | 6 | 0 | 0 | Alessio queries how Ambience evaluated complex ICD-10 medical coding. Brendan highlights the baseline 40% human physician F1 score versus their RFT-tuned o3-mini reaching 57% across 70,000 distinct codes. | |
| Managing High Compute Costs in RFT Evaluator Pipelines | 6 | 5 | 1 | 1 | Alessio brings up a war story about burning $25k on a single grader, and Wix references Braintrust's founder. Brendan discusses the balance between automated scripting, domain-expert IDEs, and ML engineers guiding intuition. | |
| Clinical Hallucinations and Out-of-Distribution Healthcare Data | 6 | 7 | 1 | 1 | Wix probes the nature of clinical hallucinations, and Brendan explains worst-case benchmark metrics and why EHR data constitutes an out-of-distribution domain. Wix connects this residency gap to insights shared by the OpenAI Codex team. | |
| HealthBench, Data Privacy, and Developing Clinical Taste | 6 | 6 | 1 | 2 | Alessio and Wix probe HealthBench, data privacy, and whether medical reasoning differs fundamentally from math/coding IQ. Brendan defines 'clinical taste' as knowing which noisy EHR elements to discard. | |
| Autonomous Clinical AI Researchers and Ambience Hiring | 4 | 3 | 0 | 0 | Alessio asks about building autonomous Devin-style clinical researchers. Brendan outlines their vision for autonomous evaluation loops and discusses the difficult archetype of clinicians with experimental engineering mindsets. |
Statements from this episode (19)
Fortuna: Ambience sells to Cleveland Clinic, UCSF, Ardent, and John Muir
“Our customers are actually, like, big health systems. So, like, Cleveland Clinic and UCSF, Ardent, Sean Muir, these are the kind of customers we sell to”
Fortuna: Ambience saves clinicians up to two hours per day
“And the end result is like, we'll save doctors, you know, up to two hours a day.”
Epic controls over 50% and Oracle Cerner holds 27% of EHR market
“Epic is obviously like the Goliath in the room, over 50% share. Cerner, that's another really big one owned by Oracle, maybe about 27%.”
Fortuna: Ambient Scribing Is Only Five Percent of AI's Healthcare Value
“That's, we listen to audio and we take notes, but that's actually just like, I would say five percent of like the total value that we can kind of offer, you know, once you have the audio of a conversation, right. And once you have access to the EHR and you can…”
Ambience AI Stack Uses Prompting, RAG, SFT, and RFT
“Ambience internally, we use prompting, we use chaining, we'll use RAG, we use fine tuning. It will use SFT and RFT.”
Fortuna: Ambience uses OpenAI's platform for reinforcement fine-tuning
“Our foray into RFT has primarily been through the OpenAI kind of platform. So we're using those self-service, you know, APIs.”
Fortuna: RFT is tremendously sample efficient compared to supervised fine-tuning
“And the second I think is like, it's tremendously sample efficient. Right. Each example sort of blooms into dozens of labels, right. And trajectories. So you can squeeze like X more signal right out of a dataset.”
Fortuna: LLM Graders for Prose Generation Are Highly Vulnerable to Reward Hacking
“And whenever using like an LLM grader, the task is like a little bit more pros or a little longer form generation. You could be very vulnerable to this. The models are super clever. They're incentivized to win, but they'll cheat and they'll do weird things.”
Fortuna: Weighting Graders 75% Accuracy and 25% Style Mitigates Reward Hacking
“So what we did is in the grader, you know, in addition to just the content and like the semantic accuracy of what it's saying, we also started to add style. And we kind of weight them like 75, 25, and over time you can kind of harness and get the reward hackin…”
RFT boosted o3-mini to 57% F1 on medical coding versus clinicians' 40%
“And the clinicians using F-one score were scoring like, let's say around 40%, right, on the F-one score, which was surprisingly low, lower than we thought. We were able to use RFT to kind of hill climb and get that, you know, get a small model O-three mini up …”
Fortuna: Oncologists and cardiologists spend up to 60 minutes pre-reviewing charts
“In certain specialties like oncology or cardiology, before they go into the visit with the patient, they can often spend like 10 minutes or up 30 minutes or 60 minutes looking at the chart. They're going to look at labs and images and, you know, other data abo…”
Fortuna: RFT on 100 examples costs thousands vs. $100 for SFT
“With like SFT, let's say you're using the OpenAI, you know, to do some supervised fine turning, you'll probably have like, you know, maybe a few thousand examples. The job takes a few hours. It costs you like a hundred bucks, right? With RFT, maybe you have li…”
Ambience Healthcare uses Braintrust for on-premise LLM observability
“One of the cool tools that we use internally is Braintrust. I think they're doing some incredible work over there, building like a tool for domain experts. They do give some built in observability. They let you deploy on premise. It's a fantastic technology. H…”
Fortuna: Domain experts cannot replace ML engineers in LLM fine-tuning
“I think the domain experts, like in our case, clinicians, they're really good at like debugging model outputs, meeting with users, distilling that feedback into something actionable, maybe annotating or doing evals, but they don't necessarily have like, you kn…”
Fortuna: Healthcare AI evaluation must measure worst-case failures over best-of-N
“I think one of the problems is, like, the benchmarks, they always report, like, the best event. So they report, like, you know, what's the best metric they got out of 64 attempts? But in healthcare, we're more interested in, like, what's the worst event, right…”
Fortuna: Base LLMs unsafely infer unconfirmed diagnoses from patient symptoms
“Patient, you know, they start to make these medical inferences. They're so smart, but they start to, like, infer things that the doctor didn't actually explicitly say. You know, for instance, a patient will say, like, I'm feeling sad and stressed out. Difficul…”
Fortuna: Real-world clinical data is out of distribution for base AI models
“I actually think clinical real world clinical data is out of distribution. I think as much as the model is generalized, if you have no access to that data, it's really hard to learn. I think that the reasons are maybe twofold. The first is a lot of realistic c…”