Feb 26, 2025 · 20m · latent-space
Raycast: Your AI Automation Assistant
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Raycast CEO Thomas Paul Mann joins the Latent Space podcast to discuss the launch of AI Extensions, explaining how the platform leverages model distillation, TypeScript developer tooling, and a proprietary model gateway to turn the desktop launcher into an AI-native operating system.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 16% of the talking time here. How this is scored →
The hosts as informed peer 5.3 Guest teaching 4.8 Guest disagreement 0.5 The hosts pushing back 0.8
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 12:46 Challenging the premise of universal API compatibility
Thomas rejects the notion that alternative models are drop-in OpenAI compatible, highlighting breaking nuances in finish reasons and tool call formatting.
Hardest push from the hosts ▶ 11:05 Swyx challenging tool selection scalabilitySwyx pushes on the scalability of extension ecosystems by pointing out that every installed extension introduces choice overhead for the LLM.
Biggest teaching moment ▶ 13:35 Breaking down the unsolved problem of tool-call evalsThomas educates the hosts on how generic LLM eval frameworks fail when applied to chained tool calling, explaining the necessity of building mock state pipelines.
The host holds their own ▶ 10:44 Swyx anticipating reasoning model latency trade-offsSwyx demonstrates domain insight by predicting that chain-of-thought reasoning models introduce prohibitive latency for OS-level launcher UI interactions.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Raycast's Journey into AI and Extensions | 4 | 3 | 0 | 0 | Swyx sets up the conversation by sharing his personal switch from Alfred to Raycast. Thomas explains Raycast's transition from an OS spotlight alternative to a global AI textbox and extensible platform in a collaborative, welcoming tone. | |
| Live Demonstration of AI Extensions and Meeting Assistant | 5 | 4 | 0 | 1 | Thomas performs a screen-share demonstration of natural language Slack status updates and meeting scheduling. Alessio probes technically on whether the architecture uses direct API OAuth rather than computer-use agents. | |
| Fine-Tuning Ray 1 and Model Distillation | 6 | 5 | 1 | 1 | Swyx notices the proprietary Ray 1 model and queries Thomas on distillation datasets and reasoning models. Thomas explains why distillation from GPT-4o to 4o-mini outperformed reasoning models for multi-step tool calls. | |
| Extension Routing and Tool Selection Management | 6 | 5 | 1 | 2 | Swyx highlights the combinatorial overhead of tool selection with dozens of installed extensions. Thomas details their @mention routing strategy, OpenAI's 128-tool limit, and the reality of model API incompatibilities. | |
| Designing Evals for Tool Calling and Developer DSLs | 5 | 7 | 1 | 0 | Alessio prompts Thomas on evaluation frameworks. Thomas provides an in-depth breakdown of why tool-calling evals are unsolved in standard tooling, necessitating custom mock infrastructures and a developer DSL. | |
| Developer Experience: Building AI Extensions with TypeScript | 6 | 5 | 0 | 1 | Alessio compares Raycast's ecosystem to Lindy and Dust, asking about background scheduling. Thomas details their TypeScript JSDoc extraction method that abstracts LLM function calling for regular software developers. |
Statements from this episode (13)
Raycast announces AI extensions allowing developers to extend Raycast AI
“So tomorrow we're going to release our next big feature, which is called AI extensions. And that basically brings together our roots of like having developers extending Raycost. And this time they can extend Raycost AI.”
Mann: Raycast AI Extensions operate via OAuth and APIs, not computer use
“So this is primarily like using like OAuth and then doing API calls and then developers basically can expose those information to our AI and then we're picking it up and composing those information together.”
Raycast built its Ray 1 models by fine-tuning GPT-4o and mini
“And so we looked into all the various models we had and then we picked, at the moment, it's gbd-for-o and gbd-for-o-mini, Which we basically did a fine tune to really optimize for our use case, and then basically shipping that in the app as Ray one and Ray one…”
OpenAI models remain the industry best for chained function calling
“And so we find like the open AI models function calling wise for our use case. So for the best ones and like, yeah, basically verifying the others. We had a beta group and testing different models. So basically from all providers and yeah, find basically the b…”
OpenAI API throws errors when users exceed a 128-tool limit
“Which OpenAI would be figured out at some point is, like, oh, there is an upper limit of, like, a 128 Tools you can have. And then the API basically gives you an error. So we ran into some of those issues as well.”
Mann: Raycast Built Its AI Model Router Internally
“And we're not using, like, a proxy, like a router or something like this, so we built that internally.”
Mann: OpenAI-Compatible APIs Still Have Subtle Behavioral Differences
“Like, even though all models say they support open AI compatible APIs, they're always in nuances which are slightly different and then put you off when you see it for the first time.”
Mann: AI evaluations for tool calling are super unsolved across industry
“Nobody talks about how to do evals with tool calls. It's, like, a super unsolved thing.”
Raycast built a DSL for third-party developers to run AI extension evals
“So we basically came up with a system that they can write like a DSL for those evals, and then they can run the evals themselves.”
Mann: Raycast extracts LLM tool definitions directly from TypeScript JSDoc
“Basically what we came up with, it's essentially you just write a TypeScript function and you document your TypeScript function with JSDoc. And then we extract all the information from there and basically make that and pass that information to the LLMs.”
Mann: All Raycast AI extensions will be open source, launching with 50
“So all of those AI extensions going to be open source. We have 50 at the beginning.”