Ruiz: Vision models effectively interpret mixed visual assets on infinite canvases
Steve Ruiz · The Accidental AI Canvas - with Steve Ruiz of tldraw · Jan 5, 2024 · at 42:52
Steve Ruiz, founder of tldraw, explains why an infinite canvas functions well as an arbitrary multimodal prompting environment for foundation models.
0:00 / 0:28exact quote · 28.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The fun of the Infinite Canvas and Teal Draw in particular is that you could just dump like whatever you want onto the canvas. Screenshots, text, images, other websites sticky notes, all that stuff. And the model, even as something that was in preview, like the very, very first sort of multimodal model can do a really good job at just taking all that stuff as the input.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Steve Ruiz
Ruiz: Drawing apps combined with generative models have no competitive moat
“There's nothing really defensible about like, Hey, it's an, it's a drawing app plus an LCM like model because there's going to be a lot of those models and there's going to be a lot of drawing apps.”
Ruiz: Collaborative whiteboards and text-to-diagram AI will become ubiquitous software commodities
“Kanban boards are in every productivity app now. I think the same thing is going to happen with collaborative whiteboards. It's like people like them. I'm making it easy. People are already doing it even without Teal Draw when it's hard. Like, yeah, that's goi…”
Ruiz: Visual software challenges stem from aesthetic taste, not technical complexity
“It wasn't necessarily like technical problems that were really hard. It was more subjective problems where I think the thing that was lacking was the taste or the opinions or the, like the feeling for what good solutions were.”
Ruiz: GPT-4V turns low-quality photos of paper wireframes into functional websites
“I can just take the screenshot here of, you know, that the dude posted of the drawing that he had made, you know, it's not even like a good photo. There's a pen, you know, across one of the screens, et cetera. But if you just give that with no other informatio…”
Ruiz: Users will never get complex AI prompts right on the first attempt
“You probably won't get ever get the prompt just right. Even if you have like a really, really, really good, you know, Three generations from now agent. Like you still have to put that information in. But you're never going to put the, all of the information in…”