The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 19 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Shah: AI models attempting medical diagnoses will kill patients
“I mean, they just keep going there. I'm going to do diagnoses because it's grand or it's challenging, but I mean, they're going to kill somebody.”
Shah: Patient non-adherence is healthcare's biggest problem, not diagnosis
“Our biggest issue in the U.S. And most of the developed world is the people don't follow the directions after they're diagnosed. Like, it's not an issue of diagnoses. It's an issue of adherence.”
Shah: Foundational LLMs are instruction-tuned for queries, not multi-turn conversations
“They've all instruction tuned for what I call a query, not a conversation.”
Shah: Operating a voice LLM costs roughly 18 cents per hour
“A large language model speaking at a hundred words per minute will cost somewhere around 18 cents an hour. Okay. So the LLM plus what's the ASR cost, the automatic speech recognition cost, the text-to-speech cost, the TTS cost. We kind of put it all in there. …”
Shah: Hippocratic AI is building its healthcare LLM from scratch
“That's why we're building our own model and we're building it from scratch.”
Shah: Hippocratic AI outperformed GPT-4 on 105 of 114 healthcare exams
“And then we took it, and then we had GPT-IV take it, and we had all the other language models take it, and we beat them all. And we beat them on a 105 of a 114 for GPT-IV, for example.”
Shah: Healthcare LLMs require tone classification to detect pain and anger
“You wouldn't need an, a tone classifier for most interactions with an LLM, but you do in healthcare. Because a lot of the information is in the anger, the pain, the frustration.”
Shah: Hippocratic AI operates fully in person five days a week
“We're a hundred percent in person. Our company is literally five days a week”
Shah: Pre-generative AI functioned as an idiot savant in narrow tasks
“It's been a bit of what I call an idiot savant till now. It could do one narrow thing well, but if you took it anything outside of that, it just like didn't work. You know, at least traditional classifier AI.”
Shah: The US healthcare system is 20% to 30% understaffed in nursing
“And so in the country today, we're 20%, 30% understaffed and like nurses alone. This isn't just true in the US. This is true in every single country out there.”
Shah: US health plans avoid prevention because Americans change jobs triennially
“It turns out the average person in the US changes healthcare plans every three years because they change jobs every three years. And so the healthcare plan will bear the cost to do the prevention, but won't basically the benefit because actually most of the be…”
Shah: Not interrupting patients is the primary driver of perceived doctor empathy
“The number one driver of whether you thought you had an empathetic doctor was did they not cut you off when you started telling your story?”
Shah: Llama 2 instruction tuning datasets averaged one instruction per sample
“The average number of instructions on, there's, they have like seven data sets they showed for their instruction tuning. The average number of instructions? One.”
Shah: Hippocratic AI acquired 1.5 trillion healthcare tokens for training
“We've actually gone in and acquired 1.5 trillion healthcare tokens.”
Shah: US health systems will buy finished AI applications, not APIs
“Most of the health systems in the US are going to need a ready finished product slash application, not just have a an API.”
Shah: Deploying in low-risk applications is the best hallucination mitigation
“The number one way to deal with hallucinations is pick the right applications.”
Shah: Roughly 6,900 of 7,000 US hospitals use standardized medical protocols
“If you look at the 7000 hospitals in the country, like, you know, 6900 will probably use the same protocol.”