Jason Liu
Developer Experience Engineer, OpenAI · 1 appearance on the record.
computed by AI from the episodes · how this works → · full disclaimer →
engineerinvestorfounderother@jxnlco ↗LinkedIn ↗jxnl.co ↗
Jason Liu is the creator of Instructor, a popular open-source library that leverages Pydantic for structured LLM data extraction and validation. Previously a Staff ML Engineer at Stitch Fix and founder of AI consultancy 567 Studio, he has trained developers globally on retrieval-augmented generation and AI system design.
27statements → 4claims → 1claims resolved → 3.63/5average certainty → 2.52/5average debate potential → 3said about them ↓
1 supported 0 partly supported 0 contradicted 3 not checkable as stated how the 4 claims stand · each chip opens the sources
3 predictions · 1 assertion · 5 opinions · 16 insights · 2 disclosures · every statement was checked. The predictions and assertion are the 4 claims: statements the public record can support or contradict. 1 is resolved, and 3 name no date, number or outcome precise enough to check. Everything else (opinions, insights, what ifs, disclosures) can never be settled by the record, so it carries no assessment.
The record, in short
What the tape says about how Jason argues and how the claims held up. Everything they said, and everything said about them, is in the tabs below.
Everything Jason Liu said on Latent Space that made the record, most notable first. Filter by type, assessment or year in the ledger →
Liu: LLM observability startups ignore full systems; just use Postgres
“The issue really is the fact that these observability companies isn't actually doing observability for the system, it's just doing the LLM thing. Like I still end up using like Datadog, right? Or like, you know, Sentry to do, like, latency. And so I just have …”
Liu: Companies abandon LLM frameworks to regain control over prompts
“So much of it is changing that if you give control of these systems away too early, you end up ultimately wanting them back. Like many companies I know that I reach out or ones were like, oh, we're going off of the frameworks because now that we know what the …”
Liu: App-layer AI startups should avoid hiring traditional machine learning engineers
“I think a lot of these app layer startups should not be hiring MLEs because they end up churning.”
Liu: XGBoost rankers will beat LLMs for large-scale tool selection
“Yeah, my money is on the rankers because you can do those so easily, right? You could just say, well, given the embeddings of my search query and the embeddings of the description, I can just train XGBoost and just make sure that I have very high, like, MRR, w…”
Liu: AI agents should use explicit DAG workflows over ReAct loops
“Instead of doing like a react type reasoning loop, I think my belief is that we should be using like workflows, right? If we do this, then we always have a request and a complete workflow. We can fine tune a model that has a better workflow. Whereas it's hard …”
Liu: LangChain and LlamaIndex can hit $100M revenue, but billions uncertain
“I think the bigger challenge is like, okay, a hundred million dollars, probably pretty, pretty easy. It's just time and effort. And they have both like the manpower and the money to sort of solve those problems. I think it's just like, again, if you go the VC …”
Liu: Fine-tuning on proprietary data at scale always beats off-the-shelf models
“We, because, I mean, at this point, we would have, like, you know, three million pieces of inventory, over, like, a billion interactions between users and clothes. Any kind of fine-taining would definitely outperform like, some off-the-shove model.”
Liu: LLM JSON mode is worse than function calling for structured outputs
“In terms of whether or not, like, JSON mode is better, I usually think it's almost worse unless you want to spend less money on, like, the prompt tokens that the function call represents. Primarily because with JSON mode you don't actually specify the schema.”
Liu: Rank and retrieve tool definitions instead of passing dozens to LLMs
“If you're running into issues where you have, like, 20 or 50 or 60 function calls, I think you're much better having those specifications saved in a vector database, and then have them be retrieved. So if there are 30 tools, like, you should basically be, like…”
Liu: Claude 3 Haiku outperforms OpenAI models at function calling
“Overall, I'm like super happy with the anthropic models compared to the OpenAM models. Like, Sonnet is very cost effective. Haiku is, in function calling, it's actually better.”
Liu: Structured LLM outputs unlock traditional computer science reasoning algorithms
“Embeddings really is kind of like the lowest hanging fruit, and using something like Instructor can really help produce a data structure, and then you can just use your computer science to reason about this data structure.”
Liu: Instructor is not a billion-dollar startup opportunity
“But to back to the instructor thing, I just don't think it's a billion dollar company.”
Liu: DSPy excels at micro-tasks but fails at end-to-end business workflows
“I think something like DSPy can work because there are like very short term metrics to measure success, right? It is like, did you find the PII or like, did you write the multi-hop question the correct way? But in these like workflows that I've been managing, …”
Liu: Data science skill sets will outvalue traditional machine learning engineering
“I think a lot more data science is going to come in versus machine learning engineering, because a lot of it now is just quantifying, like, what does the business actually want as an outcome, right?”
Liu: AI recommenders can sell $20 shirts but struggle with luxury narratives
“Narrative matters a lot to human beings. And I think the recommendation system, that's really hard to capture. Like, it's easy to sell, it's easy to use AI to sell, like, a 20 dollar shirt, but it's really hard for AI to sell, like, a 500 dollar shirt.”
Liu: Function calling decouples schemas from prompt instructions
“Function calling lets you define the schema separate from the data and the instructions. And what this meant was you can kind of have a lot more complex schemas and just map them in Pydantic, and then you can just keep those very separate.”
Liu: Instructor was designed as a minimal wrapper akin to Requests
“And so I just said, let me write, like, the most simple SDK around the OpenAI SDK, sorry, simple wrapper on the SDK, just handle the response model a bit, and kind of think of myself more like requests than an actual framework that people can use.”
Liu: Developers love building custom frameworks but hate writing JSON parsers
“People want to build their own frameworks, but people don't want to build, like, JSON parsing.”
Liu: Single schema extraction beats parallel function calling for relationship modeling
“In terms of an extraction workflow, I definitely think it's probably more helpful to have everything be a single schema. Just because you can sort of specify relationships between these entities, right, that you can't do in parallel function calling, you can h…”
Liu: LLMs excel at identifying nodes and edges for knowledge graphs
“One of the things we found out about these language models is that not only can you define nodes, it's really good at figuring out what are nodes and what are edges.”
Liu: Prefect and Zapier are well-positioned to build AI workflow UIs
“I think, you know, people like Prefect and Zapier have a pretty good shot at doing a good job.”
Liu: High-agency individuals focus on process metrics over outcome metrics
“I think the higher agency person is more focused on like process metrics versus outcome metrics. Right? Like, from pottery, like, one thing I learned was, if you want to be good at pottery, you shouldn't count, like, the number of cups or bowls you make. You s…”
The other half of the tape: Jason Liu's own voice is left out of every number here. Other people bring the name up 3 times in 3 episodes on Latent Space. every mention, with the transcript →
Who brings them up most Shawn Wang 1Alessio Fanelli 1
Every mention by year
tap a year for its mentions
2025 2 mentions in 2 episodes 1 per episode
2024 1 mention in 1 episode
Appearances (1)
| Episode | Date | Speaking time |
|---|---|---|
| High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor | Apr 24, 2024 | 37m |