Jan 5, 2024 · 1h 27m · latent-space
The Accidental AI Canvas - with Steve Ruiz of tldraw
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of Latent Space, tldraw founder Steve Ruiz recounts his journey from fine arts to software engineering and explains how his open-source infinite canvas engine became an essential spatial interface for multimodal AI prototyping.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.1 Guest teaching 2.9 Guest disagreement 0.4 The hosts pushing back 0.9
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:09:45 Rejecting the AI wrapper pivot framing
Steve firmly rejects the idea of turning tldraw into a standalone AI SaaS product, arguing that model wrappers lack defensible moats compared to underlying canvas infrastructure.
Hardest push from the hosts ▶ 46:00 Sean presses on code export limitationsSean questions the production readiness of Make It Real, pressing Steve on whether generated code is merely disposable UI or genuinely usable in modular React codebases.
Biggest teaching moment ▶ 7:45 Steve explains vector ink vs raster stampsSteve educates the audience and host on the complex computational geometry required to dynamically generate polygon vector outlines for digital ink rather than stamping raster circles.
The host holds their own ▶ 57:20 Sean connects cloud diagrams to full-stack generationSean draws upon his AWS systems experience to propose extending visual canvas prompting into cloud architecture schemas and relational database generation.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| From Fine Art to Open Source: Perfect Arrows | 3 | 2 | 0 | 0 | Sean opens with warm banter and asks Steve to explain how his fine arts background led to UI prototyping. Steve gives an extended biographical overview detailing his transition from studio art to Framer and his work on Perfect Arrows. | |
| Inventing Perfect Freehand and Canvas Interaction Mechanics | 3 | 4 | 0 | 0 | Steve explains the algorithmic challenge of creating vector-based digital ink in Perfect Freehand versus standard raster stamps. Sean listens attentively as Steve describes the physics and mathematics behind the library. | |
| Founding tldraw and Creating the Canvas Infrastructure | 4 | 2 | 0 | 2 | Steve details the genesis of tldraw, including turning down an Adobe job offer after receiving inbound enterprise contracts and GitHub sponsorships. Sean intervenes briefly to clarify whether the $200k sponsorship figure was monthly or cumulative. | |
| Make It Real: Generating Interactive Web Prototypes | 4 | 3 | 0 | 0 | Steve screenshares and demonstrates Make It Real, showing how multi-modal vision prompting converts hand-drawn UI and annotations into functional interactive web widgets. Sean appreciates the engineering alignment between the canvas architecture and LLM vision. | |
| Comparing Prompt Granularity and Paper Wireframe Ingestion | 4 | 3 | 1 | 1 | Steve runs a live benchmark comparing four variations of prompt granularity on a kitchen timer widget and shows paper wireframe ingestion. Sean notes how XState/state chart thinking fits naturally into visual prompting. | |
| Spatial Prompting, Live APIs, and System Architectures | 6 | 3 | 1 | 3 | Sean challenges the practicality of generated code, questioning whether it is limited to throwaway single-file prototypes or viable for backend and React architectures. Steve demonstrates live API integration with dog.ceo and explains how canvas metadata handles OCR limitations. | |
| Real-Time Generation with Latent Consistency Models | 4 | 2 | 0 | 0 | Steve showcases real-time diffusion rendering with Latent Consistency Models on drawfast and lens.tldraw.com. Both reflect on the intersection between fine art performance, collaborative drawing, and real-time generation. | |
| The Future of tldraw, Canvas Moats, and Founder Advice | 5 | 4 | 1 | 1 | Sean probes Steve on tldraw's commercial strategy, questioning why they do not pivot entirely into an AI wrapper company. Steve articulates the defensibility of the canvas engine itself over point AI solutions, followed by lightning-round reflections on founder intuition. |
Statements from this episode (20)
Ruiz: Interactive prototypes are necessary to define novel interactive software features
“In all those places, you kind of have to build something in order to figure out what you want to build.”
Ruiz: Visual software challenges stem from aesthetic taste, not technical complexity
“It wasn't necessarily like technical problems that were really hard. It was more subjective problems where I think the thing that was lacking was the taste or the opinions or the, like the feeling for what good solutions were.”
Ruiz: Perfect Freehand is used by Canva, draw.io, and Excalidraw
“So that was perfect freehand, and that's now used in, like, Canva uses it, like, draw.io uses it, Excaldraw uses it, we use it at Tealdraw, all over the place.”
Ruiz: Infinite canvas tools must follow standardized interaction conventions
“Like, if you're making a canvas this way, you have to kind of do them all. Like, your undo, redo should work in a specific way. Your selection should work in a specific way. Like, you know, the camera position and how the camera moves should work in a, you kno…”
Ruiz: tldraw Hit #1 on Hacker News and 40K Users on Launch Day
“There was like number one on Hacker News for a while. It was like the top trending repo on GitHub. A lot of people get, like, 40,000 people showed up at teilder.com to use it on that launch date.”
Ruiz: tldraw accumulated nearly $200K in sponsorships before raising venture capital
“By then I had had almost 200,000 dollars of sponsorship, you know, and people were just signing up and signing up because there was no way to even be a customer.”
Ruiz: tldraw's 'Make It Real' demo garnered 22 million views in 30 days
“It was like, I think we're at like, For this month, last 30 days, like twenty two million views or something like that.”
Ruiz: Infinite canvas is the right interface for multimodal AI prototyping
“Like, if I hadn't spent the last two years building this, I would spend the next two years building this. Like, it is the right product for this type of type of feature.”
Ruiz: Multimodal models don't need complex wireframes to generate UI components
“Sometimes you see with these multimodal prompting, like someone will draw a calculator, like in, in a lot of complexity and say, you know, make this real. And sure enough, you get back like a really complex full calculator. But if you did the same thing and yo…”
Ruiz: GPT-4V turns low-quality photos of paper wireframes into functional websites
“I can just take the screenshot here of, you know, that the dude posted of the drawing that he had made, you know, it's not even like a good photo. There's a pen, you know, across one of the screens, et cetera. But if you just give that with no other informatio…”
Ruiz: Vision models effectively interpret mixed visual assets on infinite canvases
“The fun of the Infinite Canvas and Teal Draw in particular is that you could just dump like whatever you want onto the canvas. Screenshots, text, images, other websites sticky notes, all that stuff. And the model, even as something that was in preview, like th…”
Ruiz: AI coding's primary challenge is prompt iteration, not generating code
“The challenge is more on the input side than the output side. Like absolutely you could figure out a way for this thing to spit out like a working iOS app or something like that. The question is like, how do you tell it what you want and how do you iterate whe…”
Ruiz: Users will never get complex AI prompts right on the first attempt
“You probably won't get ever get the prompt just right. Even if you have like a really, really, really good, you know, Three generations from now agent. Like you still have to put that information in. But you're never going to put the, all of the information in…”
Ruiz: tldraw sends text strings alongside canvas images to assist vision models
“Inside of the prompt for this, we do give it, like, an array of all the text that you've put in. We say, like, look, I know your vision isn't so good, or you have a hard time reading text sometimes when it's small... So we send those strings separately so that…”
Ruiz: tldraw's real-time generative AI demo updates in 32 milliseconds
“No, I think it's now, like, to 32 milliseconds, basically as you go.”
Ruiz: Real-time generative AI's value is the interaction experience, not outputs
“The output of this, like, while it is, like, a visual output, the output, like, doesn't actually matter. Like, it's gone in, in 16 milliseconds, and it's not coming back. And I think with all this AI stuff right now, just where we are with it, and just how com…”
Ruiz: Drawing apps combined with generative models have no competitive moat
“There's nothing really defensible about like, Hey, it's an, it's a drawing app plus an LCM like model because there's going to be a lot of those models and there's going to be a lot of drawing apps.”
Ruiz: tldraw will require commercial licenses for funded companies
“Really the only change that's gonna happen once we launch it is we're gonna start selling commercial licenses for it. So if you are using TealDraw in a commercial product, or if you want to, then you know, if it's If you're funded or if you have revenue, then,…”
Ruiz: Collaborative whiteboards and text-to-diagram AI will become ubiquitous software commodities
“Kanban boards are in every productivity app now. I think the same thing is going to happen with collaborative whiteboards. It's like people like them. I'm making it easy. People are already doing it even without Teal Draw when it's hard. Like, yeah, that's goi…”