Mar 17, 2026 · 1h 27m · latent-space
Anthropic’s Felix Rieseberg on AI Coworkers, Local-First Agents, and the Future of Knowledge Work
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Anthropic engineer Felix Rieseberg joins Latent Space to unpack the architecture, design philosophy, and broader implications of Claude Cowork, an autonomous desktop agent. He demonstrates how local-first sandboxed virtualization, simple markdown-based skills, and structured planning interfaces bridge the gap between frontier models and mainstream knowledge work.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 35.4% of the talking time here. How this is scored →
The hosts as informed peer 5.8 Guest teaching 5.0 Guest disagreement 2.0 The hosts pushing back 1.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 17:33 Silicon Valley is undervaluing the local computer
Felix firmly rejects the common AI startup thesis that all agent execution will migrate to the cloud, arguing that local machines hold critical tools and context.
Hardest push from the hosts ▶ 4:09 Swyx reframing user-friendly as power-user supersetSwyx directly challenges Felix's description of Cowork as a simplified consumer tool, asserting that its deep OS and browser integrations make it a superior power-user environment.
Biggest teaching moment ▶ 1:18:00 Why Chromium and Electron beat native WebViewsFelix educates the hosts on the reality of desktop rendering engines, explaining how WKWebView bugs strand users unless developers bundle Chromium's vast hardware workaround matrix.
The host holds their own ▶ 27:44 Swyx's autonomous YouTube ingestion and skill-orchestration pipelineSwyx demonstrates deep practical mastery by showcasing a live multi-step Cowork pipeline that visually inspects video frames and auto-generates modular skills to run his media operations.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Studio Welcome and Early Impressions of Claude Cowork | 4 | 2 | 1 | 1 | Swyx and Alessio welcome Felix to their new studio and share their enthusiastic early impressions of Claude Cowork. The atmosphere is warm, collaborative, and conversational. | |
| Claude Cowork Architecture: User-Friendly Scaffolding for Knowledge Work | 6 | 4 | 2 | 3 | Felix explains how Cowork wraps Claude Code inside a sandboxed VM for knowledge workers. Swyx pushes back on the 'user-friendly' framing, noting it acts as a power-user superset. | |
| Anthropic's Prototype Culture and Cheap Execution | 5 | 5 | 3 | 2 | Felix details Anthropic's rapid prototyping culture where building working software is cheaper than writing detailed product specs. Swyx expresses surprise at the notion that execution is now cheap. | |
| Platform Primitives vs. Hyper-Personalized Software | 6 | 6 | 4 | 2 | Felix shares a contrarian take rejecting hyper-personalized isolated software in favor of shared platform primitives. Alessio contributes his perspective on platform substrates gaining value. | |
| Sandboxed Linux Virtual Machines and Chrome Integration | 5 | 5 | 1 | 1 | Felix outlines the architectural role of Linux VMs and tight Chrome integration in giving Claude its own workspace. Swyx shares examples of bypassing complex manual authentication via Claude. | |
| Recursive Agent Execution and Remote Bug Triage | 6 | 6 | 4 | 3 | Felix describes recursively delegating crash triage to sub-instances of Claude Code Remote. When Swyx asks if everything should be cloud-first, Felix passionately defends the irreplaceable value of local machines. | |
| Benchmarking Knowledge Work and Multi-Step Planning | 5 | 5 | 2 | 2 | Swyx asks for clarification on planning and evals between Claude Code and Cowork. Felix clarifies how system prompt steering and multi-step planning differentiate non-coding knowledge benchmarks. | |
| Overcoming Model Overhang and the Genesis of Skills | 5 | 6 | 2 | 1 | Felix explains model overhang and why heavyweight scaffolding might quickly become obsolete. He illustrates this with Barry's discovery that simple markdown files outperformed complex MCP tooling. | |
| Practical Automation: Media Pipelines and Custom Skills | 7 | 3 | 1 | 1 | Swyx presents his live media production automation pipeline built on Cowork skills. Felix expresses genuine admiration for the recursive skill-splitting setup Swyx developed. | |
| Live Desktop Organization and User Interaction UI | 5 | 4 | 1 | 1 | The hosts run a live test having Cowork clean up Swyx's messy desktop while discussing user confirmation UI and progressive automation loops. | |
| Design-to-Code and Integrated Browser Workflows | 6 | 4 | 2 | 2 | Swyx shows one-shot design-to-code generation using integrated browser vision. Felix explains why meeting users in their existing browser environment beats building standalone agentic browsers. | |
| Skills Portability, Packaging, and Open Plugins | 6 | 5 | 3 | 2 | Alessio and Felix discuss skills packaging, git-backed plugins, and the tension between shared portable instructions and private user context. Felix dismisses generic flight booking demos. | |
| Synchronizing Agent Preferences and Context | 6 | 4 | 1 | 1 | Alessio outlines his internal tooling for symlinking agent skills across different directories. Felix agrees on the coworker mental model for synchronizing user preferences across agents. | |
| AI's Disruption of Software Verticals and Junior Roles | 6 | 5 | 2 | 2 | The hosts ask about vertical software vulnerability to general agents. Felix expresses deep concern about AI displacing entry-level knowledge work, prompting Alessio to propose career simulation engines. | |
| Engineering Apprenticeship Models and Takeoff Timelines | 5 | 5 | 2 | 2 | Felix highlights University of Waterloo's co-op model as an ideal framework for accelerating junior engineers. The conversation moves into society-level takeoff speeds and automated model training. | |
| Claude for Vertical Knowledge Work and Multimodal Vision | 6 | 4 | 1 | 1 | Swyx inspects Claude's multi-modal video inspection during desktop organization. Felix confirms Anthropic has dedicated engineering teams optimizing Cowork for finance and legal verticals. | |
| Computer Use Evolution and User-OS Interaction Models | 6 | 6 | 2 | 2 | Alessio asks about the intimacy and tension of agents sharing desktop real estate and VM memory. Felix reflects on why giving Claude a virtual cursor fails due to OS foregrounding limitations. | |
| Windows 95 in JavaScript and Cross-Platform Sandboxing | 6 | 6 | 2 | 1 | Swyx brings up Felix's viral Windows 95 in Electron project. Felix details how it started as an office debate proving that web runtime emulation could outperform native SaaS tooling. | |
| Enterprise Virtualization, Network Drivers, and Sandboxing | 6 | 7 | 1 | 1 | Felix gives a deep architectural breakdown of Windows WSL2 compute systems and Apple virtualization framework, explaining custom network drivers written to bypass corporate packet inspection. | |
| Developer Risk Tolerance and Granular Safety Controls | 6 | 5 | 2 | 2 | The panel discusses developer risk tolerance and approval fatigue. Felix argues that VMs enable safe autonomous delegation without forcing users to audit every single bash command. | |
| The Long-Horizon Roadmap of Claude Cowork | 6 | 4 | 1 | 2 | Felix outlines the long-term vision of Cowork: expanding temporal task horizons and local computer agency. Swyx proposes background continuous monitoring of daily workflows. | |
| Electron Architecture and Chromium as the Universal Operating System | 7 | 8 | 3 | 2 | Swyx asks why Electron remains necessary over Tauri. Felix delivers a masterclass on Chromium's unmatched GPU driver workarounds and the severe limits of OS-bundled WebViews. | |
| Towards Fully Native AI Code Generation and UltraThink | 7 | 6 | 2 | 3 | Alessio presents his PIQ architecture for distributed agent tasks while Swyx presses Felix on UltraThink and Anthropic Labs. The episode ends on high praise for Claude Cowork. |
Statements from this episode (52)
Swix uses Claude Cowork to manage Latent Space video uploads and edits
“I've been using it a lot, even for managing latent space, like co-work helps me upload videos and like title things and like edits and everything.”
Rieseberg: Cowork's most impressed users find unexpected capabilities
“Every single person who's like most amazed is usually amazed about a thing that I didn't even expect Cowork would be good at.”
Rieseberg: Claude Cowork is Claude Code running in a sandboxed virtual machine
“Cowork is cloud code running in a virtual machine with a little bit of padding, a little bit more guardrails, making it a little safer, a little bit more convenient for people who don't want to first open up the terminal when they go to work.”
Rieseberg: Claude Cowork's extensibility mirrors the key driver of VS Code's success
“My personal belief is that the hackability and the extendability has like played a pretty big role, right? You can hook in Visual Studio Code to like almost any workload. It's so easy to hack on, so easy to build extensions for it. And I think cold work might …”
Rieseberg: Claude Cowork assembled existing prototypes rather than starting from scratch
“And what cowork actually became is like, we sort of picked the right pieces out of the many prototypes that we had. Right. And that's maybe also like, I think an important qualifier whenever people mention this like 10 day number, I do think it's important to …”
Rieseberg: Anthropic builds all prototype candidates quickly instead of writing memos
“We internally at Anthropica are now probably much closer to the point where, like, don't even write a memo. Just, like, build, like, let's build all the candidates very quickly. Let's just build all of them and then pick the best ones.”
Fanelli: Platform substrates become more valuable as software creation costs fall
“When people are saying, oh, the value of a lot of software is going to zero because you can recreate it. To me, it's almost like the opposite. It's like having an existing platform to build on top of is like even more valuable because they can kind of bolt thi…”
Rieseberg: Future of software will not be hyper-personalized individual apps
“I actually don't think that the future is going to be hyper personalized software down to the point where everyone is running their own version. Like, I actually think it's going to be quite hard for one of us to have our own internal chat tool.”
Anthropic: Claude Cowork executes inside a dedicated lightweight Linux VM
“So we currently run like a, we currently run like a lightweight VM and we put clock code into the VM and we do that for a number of reasons. Safety and security is a big one, but even if you ignore for a second safety and security and you're just like, okay, Y…”
Rieseberg: Aggressively anthropomorphizing Claude improves agent UX and architecture design
“And in terms of architecture and UX and everything else that we've been working on Anthropic, it often is quite useful for you to like anthropomorphize cloud aggressively and just be like, this is a person. What would you do if you give, if you had a person, r…”
Rieseberg: Users reject MCP connectors due to auth friction and limitations
“You're noticing that a lot of people, especially as the models get better, a lot of people throw up their hands when it comes to MCP connectors and they say, I'm not gonna go through like 25 MCP connectors, click auth everywhere, and then like half of them don…”
Rieseberg: Anthropic did not build Claude Cowork primarily for coding
“I caught myself, like, starting to use Cowork for coding tasks, which is not ostensibly what we built it for, right?”
Rieseberg automates bug triage and dispatch using Claude Cowork loops
“The way I'm using it is I have co-work running, and I'm telling co-work, here's where I normally go every morning to find the latest bugs. Go read the entire bug list, separate out which ones are fixable, which ones are not fixable, and then for the fixable on…”
Rieseberg: Silicon Valley is undervaluing the local computer
“So I generally believe that Silicon Valley overall is undervaluing the local computer.”
Rieseberg: AI agents need parity with all user tools
“I think that entity needs to have access to all the same tools you have access to. Otherwise it's going to be hamstrung, like all these complex ways.”
Anthropic evaluates Claude Cowork on finance and legal tasks, not coding
“So like clock code is like quite optimized for coding tasks, and we mostly evaluate whether or not we're getting better or worse, depending on how good it is at like a typical sweet job. And cloud co-work on the other hand, we evaluate more against typical kno…”
Rieseberg: Unclear if agent hyper-optimizations remain relevant in next-gen models
“Will those gaps still exist in the next few generations of models? It's like a little unclear to me though. Because right now these like hyper optimizations we make, I'm not sure for how long they're still really relevant.”
Anthropic prompts Claude Cowork to clarify user intent before long executions
“We do tell co-work to make heavy use of the planning tool or to make heavy use of the ask user question tool, right? We do want it to come up with like, Different scenarios of, okay, tease out what the user actually wants. Don't go off to work for like four ho…”
Rieseberg: Specialized AI wrapper apps won't survive as models generalize
“I think we're going to see a lot of like applications and companies that do very impressive things with AI that in the short term might seem very effective because they're very specialized to individual use cases. But I think once models get better at generali…”
Rieseberg: Anthropic's skills feature originated from a simple markdown API prompt
“The thing that ultimately led to skills is that we wanted to connect this little prototype to our data warehouse. And the team very quickly discovered that like, instead of building a custom tool for the thing to talk to our data warehouse, They just, like, ma…”
Rieseberg: Claude Cowork skills can be as simple as a text message
“One thing that is very fun for me about skills in particular is that they're so easy to make. Like anyone can make a skill, like a text message could be a skill and they can be so hyper-personalized to you.”
Swix: Users adopt Claude Cowork by progressively automating click-heavy knowledge tasks
“Basically, I think like the way that people ease into cloud co-work is like take a knowledge work task that you would normally be clicking around for, and then try to turn, turn that. And then you do the, okay, well, what if you went further? Okay. And then wh…”
Rieseberg: Personal AI agent automation is like playing Factorio
“It's like a little bit like playing Factorio, but for your own life. Like you say, you start really small, you start automating something really tiny. And like, once it clicks, you keep adding onto this like automation empire, just like make your life easier a…”
Rieseberg: Claude Cowork prompt automatically prioritizes meetings with Dario Amodei
“I've given it like pretty clear instructions about, okay, here are some people, if they book over other meetings, I'm probably going to go to their meeting. Like if Dario schedules a meeting. Not try to reschedule Dario.”
Rieseberg: Desktop cleanup was one of the most viral launch use cases for Cowork
“Right when we launched Cowork, I think one of the users that went most viral on Twitter X was clean up your desktop, which is of course silly. That's such a smart thing, right? Like you don't need a model to clean up your desktop.”
Rieseberg: Giving Claude DOM and visual feedback makes coding far more effective
“And I think giving cloud eyes into like what you're actually working on makes it so much more effective. And that's probably what you've seen in code, because it can see Chrome, it can like debug the DOM, it can like see things that does make it more powerful.”
Rieseberg: Claude Code and Cowork share unified GitHub-installable plugin format
“We do have skills as part of this container format, which was just called plugins. And plugins are available both for Cloud Code and Cloud Cowork, the same format. And you can install plugins. This works in Cowork today. You can basically say, I'm going to add…”
Rieseberg: AI agents should not be booking flights
“I don't actually think AI should be booking flights.”
Fanelli: Personal productivity advantage will come from agent skills, not base AI products
“That almost feels like my personal productivity thing will be my skills. Yeah. It's not really the product that I use because everybody has access to the same product.”
Rieseberg: A 'Dropbox of skills' startup for cross-agent sync may not be viable
“I'm not sure what the right thing is going to be. If it's going to be some company that can say, all right, we're as an independent body, we're not trying to like push into any particular product. It's our job to be like the skill authority and we provide I do…”
Anthropic deeply worries about AI automating entry-level junior jobs
“At Anthropic as a group of people, we're deeply worried about the impact that the tools are going to have on the labor market, especially for like junior employees that, because I think it's only honest to say that when we talk about automating a lot away, a l…”
Fanelli: AI can simulate junior jobs to accelerate engineer training
“I think now we can use AI and these models to actually like shortcut these careers and almost like simulate the early years of your work and like just make them like super dense in like these learnings.”
Rieseberg: AI takeoff will create an accelerating, self-reinforcing loop
“Big bang moment where things will accelerate so quickly that it becomes a self-reinforcing loop. And at that point it's sort of like off to the races and there will be no more like slowly catching up. You know, just have Claude being so good at everything.”
Rieseberg: Prompt Opus by stating goals, not specifying exact execution steps
“Honestly though, like I see that you're using Opus 4.6, right? Like my recommendation for people is increasingly don't worry about it anymore. Just like tell it what you want it to do. And it's probably going to figure out a way to do it.”
Rieseberg: Claude is close to effectively controlling real user computers
“I don't think we're far away from claw being very effective at like using your computer and not just a theoretical computer.”
Rieseberg: Dual-cursor AI desktop concepts fail due to OS architecture
“We could do a second cursor, but that actually breaks down quite a bit. Even if you go and like present cool dreams to both Apple and Microsoft, you're like, wouldn't it be cool if it breaks down quite a bit because so many of our models on a computer are buil…”
Rieseberg: Claude's VM takes less than 10 gigabytes on macOS disk
“A lot of people write me like, hey, how, how come Cloud is taking up 10 gigabytes? I could say on that point, it's not actually taking up 10 gigabytes. It's just like a way that Mac OS displays bytes is like wrong, but the way we actually write it to disk is b…”
Rieseberg: Claude's Windows VM runs on Windows Host Compute System
“What we're doing on Windows, we're using the Windows, Windows host compute system. It's the same thing that WSL two runs on, like the Windows subsystem for Linux.”
Rieseberg: Anthropic is building custom network drivers for Claude
“Last week we started writing a new networking service and networking driver. That optimizes how Claw talks to the internet.”
Rieseberg: Claude Cowork uses Apple Virtualization framework on macOS
“So on, on macOS, we use the Apple virtualization framework, which is pretty solidly optimized. Like it's good stuff.”
Rieseberg: Sandboxing bypasses need for 100% model alignment
“I don't think we need to wait for like a hundred percent model alignment. We can rely on the same Swiss cheese model we've used in the industry for a long time, but I do think we need to like universally, maybe eventually invest more. And that's what we're doi…”
Rieseberg: Developers tolerate security risks trusting they can fix failures
“And I think also as developers maybe we're, we are more risk tolerant, but we're also just like accepting, we are more risk tolerant, but I think we also just have like, I don't want to say arrogance, but like sort of the trust that if like the really bad thin…”
Rieseberg: Users will not scrutinize AI-generated scripts in daily workflows
“Because I kind of believe that Once it starts becoming a part of your workflow, you're probably not either, either you don't have the skill to understand whether or not this Python script is safe, or you're not going to read it anyway.”
Rieseberg: Claude Cowork will release new features weekly
“We're going to keep shipping things that we're going to keep shipping things that we're going to keep iterating on this thing like pretty quickly, but which I mean, you can sort of continue to expect that every single week, there's going to be like a small new…”
Rieseberg: Engineers get maximum leverage by investing in interface layers
“Whenever I give other engineers advice, it's actually that layer that I think is most valuable to invest in, because the tools of the layer are not that good, but that's where you get the most leverage.”
Rieseberg: Desktop apps need embedded Chromium because OS WebViews are inadequate
“The reason to go with your own embedded rendering engine is because, and this is still true in 20, 26, the operating system rendering engines are not that good. They're just not that good. Both Microsoft and Apple are trying to move away from that. They so far…”
Rieseberg: Current AI models can replicate Electron apps in Swift
“I think our current models are quite capable of taking an Electron app and replicating it Swift.”
Rieseberg: Anthropic built tech enabling Claude to interact via Google Doc comments
“We built so much tech around Claude leaving useful comments inside a Google Doc, and now it just does it, just like leaves a comment in your Google Doc and that's how you interact with it.”
Fanelli: Future AI coworkers will queue cross-departmental decisions for human approval
“And to me, it's almost like in the organization of the future, it's like the salespeople are talking to the engineering team that is talking to the marketing team, to the product team, and all these co-workers are going to like queue up decisions for other peo…”
Rieseberg: Building powerful software often borders on building something creepy
“This maybe goes back into the territory of like building something very powerful and building something creepy often goes hand in hand.”