Apr 24, 2024 · 1h 2m · latent-space
High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Latent Space, Jason Liu, creator of Instructor, joins Alessio and Swix to discuss the mechanics of structured LLM outputs, the architectural advantages of pragmatic engineering over VC-backed frameworks, and the emerging discipline of AI engineering.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 5.6 Guest teaching 4.8 Guest disagreement 3.2 The hosts pushing back 1.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 58:41 Startups should stop hiring PyTorch ML engineers
Jason forcefully dismisses the common startup hiring practice of recruiting PyTorch MLEs for application-layer products, pointing out that they inevitably churn when forced to fix TypeScript errors.
Hardest push from the hosts ▶ 45:34 Pushback on measuring process volume over outcomesSwix directly confronts Jason's pottery clay metaphor by arguing that measuring process volume in software engineering is the equivalent of counting lines of code.
Biggest teaching moment ▶ 25:37 Embeddings fail on temporal and aggregation queriesJason walks through explicit counterexamples showing why semantic vector search fails on date-relative or group-by queries, demonstrating the necessity of structured data extraction.
The host holds their own ▶ 19:08 Real-world incubation data on 780 API endpointsAlessio counters theoretical tool retrieval suggestions by sharing concrete empirical data from a 780-endpoint integration startup where full-context LLM selection outperformed vector ranking.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Stitch Fix Experience and Early LLM Skepticism | 6 | 5 | 2 | 2 | Swix connects RAG to classical recommendation systems (RecSys) and probes why Jason built the Flight recommendation system from scratch at Stitch Fix. Jason explains the operational necessity of standardizing bespoke data science codebases to enable observability and performance tuning. | |
| Similarity Search, Multimodal Embeddings, and Real-World Impact | 6 | 5 | 1 | 2 | Swix asks technical clarifying questions about whether off-the-shelf GPT-3 embeddings worked for multi-modal clothing retrieval. Jason clarifies that fine-tuning ResNets and joint text-image representations was necessary to beat generic models at scale. | |
| Fashion, Human Narrative, and Recommender Limitations | 3 | 6 | 3 | 1 | Swix jokes about tech fashion and asks whether tech workers should use Stitch Fix. Jason reframes the premise by contrasting utility-driven clothing delivery with human narrative and experiential shopping. | |
| The Inception and Architectural Philosophy of Instructor | 6 | 6 | 3 | 1 | Swix and Jason discuss library architecture, contrasting broad frameworks like Django with lightweight standard utilities like Requests. Jason details how function calling separated schemas from prompts, driving the inception of Instructor. | |
| Typed Outputs, JSON Mode, and Function Calling Mechanics | 7 | 5 | 3 | 2 | Alessio asks technical questions comparing JSON mode to function calling and inquires about SDK-level handling of parallel tool invocation. Jason explains why typed schemas with validation outperform raw JSON output modes. | |
| Tool Selection, Context Scaling, and Ranking Architectures | 8 | 4 | 3 | 5 | Alessio pushes back on relying solely on vector similarity for tool selection, sharing firsthand incubation data on 780 API endpoints where LLMs select tools better than semantic search. Jason defends ranking via XGBoost over stuffing prompts with tool definitions. | |
| Production Benchmarks and Anthropic vs OpenAI Trade-offs | 6 | 5 | 4 | 1 | Alessio notes Jason's contrarian evaluation of Anthropic function calling benchmarks. Jason shares real production telemetry, revealing edge-case parsing bugs despite compelling pricing advantages. | |
| Mapping the Use Cases and Surface Area of Instructor | 5 | 8 | 4 | 1 | Swix invites Jason to delineate the capabilities of Instructor across graphs, sanitation, and query understanding. Jason schools the audience on why vector embeddings fail on temporal aggregations and relational queries. | |
| Workflows and Deterministic DAGs versus ReAct Agent Loops | 6 | 5 | 4 | 2 | Swix asks how Instructor compares with LangChain's self-querying and prompts Jason on workflow design. Jason advocates for deterministic DAGs and fine-tuning compilers over non-deterministic ReAct prompting loops. | |
| Autonomous Agents, Execution Horizons, and Plan Churn | 6 | 4 | 4 | 3 | Swix discusses evaluating long-horizon coding agents like Devin and cautions against rewarding runtime duration. Jason mocks early agent failure modes and argues for monotonic progress metrics over loop thrashing. | |
| AI Engineering Stack and Pragmatic Infrastructure | 6 | 4 | 5 | 2 | Swix inquires about Jason's preferred AI engineering stack. Jason dismisses the dozens of VC-backed LLM observability startups, explaining that a single Postgres table and standard APM tools like Datadog and Sentry are more than sufficient. | |
| Choosing Independent Consulting over Venture Capital Backing | 5 | 4 | 4 | 1 | Swix asks why Jason declined venture backing for Instructor. Jason explains his philosophy of building a standard utility like Requests while running a high-margin, flexible solo consulting practice. | |
| Startup Realities, Solo Founding, and Entrepreneurial Courage | 6 | 4 | 3 | 1 | Alessio and Swix discuss the realities of seed funding versus high-paying corporate employment. Jason highlights founder breakup rates and emphasizes having the courage to build with minimal overhead. | |
| Overcoming Career Setbacks and Starting from Zero | 4 | 4 | 2 | 1 | Alessio asks about overcoming early rejection after Jason was turned down by top AI labs in early 2023. Jason uses a Jiu-Jitsu analogy to urge engineers to start immediately rather than waiting for ideal conditions. | |
| High Agency, Process Metrics, and the Clay Metaphor | 6 | 6 | 4 | 6 | Jason defines agency as focusing on process volume rather than final outcome metrics using a pottery clay metaphor. Swix directly pushes back, asking whether this equates to flawed developer metrics like counting lines of code. | |
| Automating Personal Workflows and Future AI Capabilities | 5 | 4 | 2 | 1 | Alessio and Jason discuss personal workflow automation. Jason explains how hiring human assistants helped him uncover precise, modular heuristic prompts for calendar management and client summaries. | |
| DSPy, Prompt Optimization, and Business-Level Metrics | 6 | 6 | 4 | 2 | Swix raises DSPy and the thesis that AI is a better prompt optimizer than humans. Jason argues that DSPy works well on narrow programmatic heuristics but breaks down when evaluating long-term business outcomes like user churn. | |
| AI Engineers vs ML Engineers and Organizational Talent | 6 | 5 | 5 | 1 | Swix asks about the distinction between MLEs and AI engineers. Jason delivers a strong take advising application startups against hiring traditional MLEs, who quickly churn when faced with TypeScript and missing training pipelines. | |
| AI Engineer World's Fair Preview and Episode Conclusion | 4 | 2 | 1 | 1 | Swix previews the AI Engineer World's Fair. Jason jokes about his talk title while reinforcing the core philosophy behind Pydantic-based structured outputs. |
Statements from this episode (27)
Liu: Stitch Fix used Transformer models for recommendations before GPT-3
“We actually were using Transformers at Stitch Fix, like, before the GPT-III model, so we were just using Transformers for recommendation systems.”
Liu: Fine-tuning on proprietary data at scale always beats off-the-shelf models
“We, because, I mean, at this point, we would have, like, you know, three million pieces of inventory, over, like, a billion interactions between users and clothes. Any kind of fine-taining would definitely outperform like, some off-the-shove model.”
Liu: Admitted being bearish on LLMs for four years before recent breakthroughs
“I mean, the biggest one really was the fact that, like, I think for just four years I was so bearish on language models. And just NLP in general, I was just like, ah, like, none of this really works. Like, why would I spend time focusing on this? I gotta go do…”
Liu: AI recommenders can sell $20 shirts but struggle with luxury narratives
“Narrative matters a lot to human beings. And I think the recommendation system, that's really hard to capture. Like, it's easy to sell, it's easy to use AI to sell, like, a 20 dollar shirt, but it's really hard for AI to sell, like, a 500 dollar shirt.”
Liu: Function calling decouples schemas from prompt instructions
“Function calling lets you define the schema separate from the data and the instructions. And what this meant was you can kind of have a lot more complex schemas and just map them in Pydantic, and then you can just keep those very separate.”
Liu: Instructor was designed as a minimal wrapper akin to Requests
“And so I just said, let me write, like, the most simple SDK around the OpenAI SDK, sorry, simple wrapper on the SDK, just handle the response model a bit, and kind of think of myself more like requests than an actual framework that people can use.”
Liu: Developers love building custom frameworks but hate writing JSON parsers
“People want to build their own frameworks, but people don't want to build, like, JSON parsing.”
Liu: LLM JSON mode is worse than function calling for structured outputs
“In terms of whether or not, like, JSON mode is better, I usually think it's almost worse unless you want to spend less money on, like, the prompt tokens that the function call represents. Primarily because with JSON mode you don't actually specify the schema.”
Liu: Single schema extraction beats parallel function calling for relationship modeling
“In terms of an extraction workflow, I definitely think it's probably more helpful to have everything be a single schema. Just because you can sort of specify relationships between these entities, right, that you can't do in parallel function calling, you can h…”
Liu: Rank and retrieve tool definitions instead of passing dozens to LLMs
“If you're running into issues where you have, like, 20 or 50 or 60 function calls, I think you're much better having those specifications saved in a vector database, and then have them be retrieved. So if there are 30 tools, like, you should basically be, like…”
Liu: XGBoost rankers will beat LLMs for large-scale tool selection
“Yeah, my money is on the rankers because you can do those so easily, right? You could just say, well, given the embeddings of my search query and the embeddings of the description, I can just train XGBoost and just make sure that I have very high, like, MRR, w…”
Liu: Claude 3 Haiku outperforms OpenAI models at function calling
“Overall, I'm like super happy with the anthropic models compared to the OpenAM models. Like, Sonnet is very cost effective. Haiku is, in function calling, it's actually better.”
Liu: LLMs excel at identifying nodes and edges for knowledge graphs
“One of the things we found out about these language models is that not only can you define nodes, it's really good at figuring out what are nodes and what are edges.”
Liu: Structured LLM outputs unlock traditional computer science reasoning algorithms
“Embeddings really is kind of like the lowest hanging fruit, and using something like Instructor can really help produce a data structure, and then you can just use your computer science to reason about this data structure.”
Liu: AI agents should use explicit DAG workflows over ReAct loops
“Instead of doing like a react type reasoning loop, I think my belief is that we should be using like workflows, right? If we do this, then we always have a request and a complete workflow. We can fine tune a model that has a better workflow. Whereas it's hard …”
Liu: Prefect and Zapier are well-positioned to build AI workflow UIs
“I think, you know, people like Prefect and Zapier have a pretty good shot at doing a good job.”
Liu: LLM observability startups ignore full systems; just use Postgres
“The issue really is the fact that these observability companies isn't actually doing observability for the system, it's just doing the LLM thing. Like I still end up using like Datadog, right? Or like, you know, Sentry to do, like, latency. And so I just have …”
Liu: Instructor is not a billion-dollar startup opportunity
“But to back to the instructor thing, I just don't think it's a billion dollar company.”
Liu: High-agency individuals focus on process metrics over outcome metrics
“I think the higher agency person is more focused on like process metrics versus outcome metrics. Right? Like, from pottery, like, one thing I learned was, if you want to be good at pottery, you shouldn't count, like, the number of cups or bowls you make. You s…”
Liu: Agency drives machine learning experiment volume; experience filters wasteful trials
“So, agency lets you sort of capture the volume of experiments, and, like, experience lets you figure out, like, oh, that other half, it's not worth doing.”
Liu: Define explicit conditions for revisiting negative machine learning experiment results
“Like what you should write down is like, here are the conditions. This is the inputs and the outputs we tried the experiment on. And then one thing that's really valuable is basically writing down under what conditions would I revisit these experiments?”
Liu: GPT-4 and Claude 3 Opus still write poor quality essays
“Or those are two sort of systems that I wish you before or Opus was actually good enough to just write me an essay, but most of the essays are still pretty bad.”
Liu: Companies abandon LLM frameworks to regain control over prompts
“So much of it is changing that if you give control of these systems away too early, you end up ultimately wanting them back. Like many companies I know that I reach out or ones were like, oh, we're going off of the frameworks because now that we know what the …”
Liu: LangChain and LlamaIndex can hit $100M revenue, but billions uncertain
“I think the bigger challenge is like, okay, a hundred million dollars, probably pretty, pretty easy. It's just time and effort. And they have both like the manpower and the money to sort of solve those problems. I think it's just like, again, if you go the VC …”
Liu: DSPy excels at micro-tasks but fails at end-to-end business workflows
“I think something like DSPy can work because there are like very short term metrics to measure success, right? It is like, did you find the PII or like, did you write the multi-hop question the correct way? But in these like workflows that I've been managing, …”