Apr 25, 2023 · 53m · no-priors
No Priors Ep. 7 | With Stanford Professor Dr. Percy Liang
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Stanford Professor Dr. Percy Liang discusses the paradigm shift of foundation models, the evolving dynamics between academia and industry, and the necessity of decentralized compute, multi-metric benchmarking, and transparent AI governance.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.8 Guest teaching 3.6 Guest disagreement 0.7 The hosts pushing back 0.8
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 23:15 Liang rejects straightforward emergence predictions
Liang resists the host's prompt to predict emergent capabilities, pushing back on the premise by noting how much of modern model behavior stems from intentional tuning rather than purely spontaneous emergence.
Hardest push from the hosts ▶ 37:20 Host restates scientific fraud detection thesisWhen Liang pivots to general student plagiarism, the host interrupts to clarify and insist on her original framing of detecting literature fraud and inconsistent biomedical data.
Biggest teaching moment ▶ 14:05 Liang breaks down semantic parsing vs LLMsLiang gives an authoritative technical breakdown of computational semantics, explaining why mapping language to SQL fell short in an unstructured world and how that prompted the creation of SQuAD.
The host holds their own ▶ 10:39 Host cites 1970s MYCIN expert system historyThe host demonstrates deep historical context in AI by referencing Stanford's MYCIN project from 50 years prior to challenge naive assumptions regarding medical AI adoption speed.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Dr. Percy Liang's Background in Machine Learning and NLP | 5 | 4 | 0 | 0 | The host asks an informed question citing Liang's prior career in semantic parsing. Liang explains the paradigm shift from task-specific HMMs and parsing systems to general in-context learning with GPT-3. | |
| Stanford CRFM Mission and the Retreat of Open AI Culture | 4 | 3 | 1 | 1 | The host probes the diagnosis behind industry's shift toward closed API access. Liang outlines how capital intensity, commercial moat, and emerging safety risks led companies to retreat from open research. | |
| The Diverging Roles of Academia and Industry in AI | 5 | 4 | 1 | 1 | The host asks how academic and industry roles diverge as industry scales compute. Liang clarifies that academia must shift from making models work to understanding their inner workings and measuring broader societal impacts. | |
| AI Deployment in Healthcare and Superhuman Standards | 7 | 2 | 1 | 2 | The host demonstrates strong domain depth by citing Stanford's 1970s MYCIN expert system and challenging human doctor benchmarks. Liang agrees, arguing AI evaluation should focus on superhuman statistical rigor rather than merely mimicking humans. | |
| Computational Semantics, Question Answering, and Symbolic AI | 5 | 5 | 0 | 0 | Liang delivers a masterclass on semantic parsing, mapping natural language to SQL, the SQuAD benchmark, and the neural versus symbolic debate in AI reasoning. | |
| Research Horizons Beyond Scaling: Efficiency and Complex Problem Solving | 4 | 4 | 0 | 0 | Liang explores research frontiers beyond scaling, highlighting data efficiency, longer context windows, iterative search, and emergent cross-domain creativity like explaining Quicksort in Shakespearean style. | |
| Emergent Instruction Following and Mitigating Hallucinations | 4 | 4 | 2 | 1 | Liang resists simple hype predictions, noting the ambiguity between emergent behavior and human fine-tuning. The host frames hallucination mitigation around scale and statistical accuracy, which Liang qualifies with pre-training world models. | |
| Alternatives to Transformer Architectures and Compute Bottlenecks | 6 | 3 | 1 | 2 | The host brings in an external counter-argument regarding architectural lock-in around Transformers due to compute costs. Liang explains how Transformers strictly dominate alternatives like LSTMs under fixed compute budgets and highlights non-attention architectures. | |
| Together AI: Decentralized Compute and Specialized Models | 5 | 4 | 0 | 0 | The host connects Together AI's decentralized compute approach to historical distributed systems like Folding@home. Liang details the network interconnect challenges and the vision of diverse, specialized open foundation models. | |
| Domain-Specific Models and AI-Assisted Scientific Research | 6 | 3 | 2 | 3 | The host pitches using PubMed-trained models to detect scientific fraud and image manipulation. Liang redirects toward literature synthesis and paper review, but the host politely interrupts to double down on fraud detection as a compelling angle. | |
| The Vision of an Autonomous AI Scientist | 4 | 4 | 1 | 1 | Liang lays out his vision of an autonomous AI scientist forming hypotheses and running experiments. The host connects this to Daphne Koller's work at insitro on assisted data generation and discovery loops. | |
| Holistic Evaluation of Language Models (HELM) | 4 | 4 | 0 | 0 | Liang provides a detailed breakdown of HELM, evaluating 30 models across 42 scenarios and metrics like calibration, robustness, fairness, and security vulnerabilities like jailbreaks. | |
| Policy, Governance, and Transparency Norms for AI | 4 | 3 | 0 | 0 | The host highlights CRFM's unique policy intersection. Liang advocates for transparency norms, questioning who determines alignment values and comparing release requirements to nutritional labels. | |
| Open-Source AI and Together's OpenChatKit | 4 | 3 | 1 | 0 | Liang explains Together's OpenChatKit and its collaborative open-source philosophy. When asked about AGI, he rejects dogmatic worldviews in favor of approaching emerging capabilities with 'no priors.' |
Statements from this episode (20)
Liang: Foundation Models Cause the Concept of an AI Task to Dissolve
“And this was a paradigm shift, in my opinion, because it changed the way that we conceptualize Machine learning and NLP systems from these bespoke systems where you're, it's trained to do question answering, to train to do this, to just a general substrate whe…”
Liang: The term 'Foundation Models' was coined because LLMs missed multimodality
“We coined the term Foundation Models because we thought it, there was something that was happening in the world that was, that somehow large language models didn't really capture the significance. And it was not just about language, it was about images and mul…”
Liang: The AI industry is retreating from its open culture
“And what we're seeing now is sort of a retreat of that open culture where models are now being only accessible via APIs. We don't really know all the secret sauce that's going behind them, and there's sort of limited access.”
Liang: Future AI models may be too powerful for unrestricted release
“I think these models are extremely powerful and maybe the models right now I think are, well, if they were out and open, it would be maybe okay, but in the future these models could be extremely good and having them, you know, anyone, anything goes. Might we m…”
Liang: Big tech scale forced AI academia to focus on understanding models
“And now today I think it's the dynamic is, is quite different because it's no longer academia's job isn't just to get things to work because you can do that in other ways. There's a lot of resources going into big tech companies where there's if you have data …”
Liang: AI benchmarks should target superhuman reliability over human mimicry
“I think we're getting to a point where along many axes, it's a superhuman or should be superhuman. And I think we should maybe define more of an objective measure of like what we actually want. We want something that's very reliable, is grounded. You know, I o…”
Liang: Language models should use calculators instead of computing internally
“There are cases where you want to just map natural language into say people call it tool use. Like you ask some question that reverse calculation, you should just use a calculator rather than trying to sort of quote unquote do it in the transformers head.”
Liang: Core symbolic AI problems like planning are relevant again
“There's neural versus symbolic. For a while, symbolic AI was dominant. Now, neural AI has come really taken off and become dominant, but some of those central problems of how do you do planning, how do you do reasoning, which was the focus and study of symboli…”
Liang: LLMs are not just memorizing because novel concept fusion requires invention
“You know, people say that sometimes all language models just memorize because they're so big and train on clearly a lot of texts, but these examples, I think really indicate that there's no way That these language models are just memorizing because this text j…”
Liang: Next-token prediction forces language models to build world models
“If you think about predicting the next word, It's, it seems very simple, but you have to really internalize a lot of what is going on in this context. What are the previous words? What's the syntax? What's who's saying them? And all of that information and con…”
Liang: Controlling AI hallucination is easier once models understand the concept
“So I think there's pre-training, which is predicting the next word and developing a world model, so to speak. And with those capabilities, then you can, you still have to say don't hallucinate, but it will be much easier to control that model if it has a notio…”
Liang: Stanford researchers developed attention-free architectures competitive with transformers
“So one of my colleagues, Chris Ray and his students have developed other architectures, which are actually at smaller scales, competitive with transformers. And actually don't require the central operation of attention.”
Stanford and MosaicML's BiomedLM Achieved State-of-the-Art on the USMLE
“We've trained a model here at CRFM in collaboration with Mosaic and called Biomed LM. It's not a huge model, but it's trained on PubMed articles. And it exhibits you know, pretty good, you know, performance on various benchmarks for a while you know, we were a…”
Liang: Google beat Stanford's medical AI with a model 200x larger
“Google did come up with a model that was, I think, 200 times larger and they beat that model.”
Liang: AI with human oversight can produce class-project-level research
“I think you're at the level where it could probably generate things and, you know, I think it would still be a lot of, you know, human loop, but you could generate probably let's say I don't know, a class project type of project.”
Liang: Stanford's HELM evaluated 30 models across 42 scenarios and seven metrics
“So overall, there were 30 different models, 42 scenarios and seven metrics, and we ran the same evaluations on, on all of that.”
Liang: Interconnected AI accepting external inputs risks cascading jailbreak exploits
“If these models start interacting with the world and accepting external inputs, now you can not only just sort of jailbreak your own model, but you can jailbreak other people's model and get them to do various things. And then, so that could lead to sort of a …”
Liang: AI model producers should provide transparency analogous to nutrition labels
“The analogy I like to think about is, you know, nutrition labels or any sort of specification sheets on electronic devices. There's some sort of, ah, obligation. I think that you know, producers of some products should have to make sure that their product is u…”