Mar 9, 2023 · 39m · a16z
Unlocking Creativity with Prompt Engineering
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the a16z Podcast, host Steph Smith interviews AI pioneer Guy Parsons to unpack the foundational principles, evolving workflows, and future career potential of prompt engineering across generative image models like Midjourney, DALL-E 2, and Stable Diffusion.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The host holds 46% of the talking time here. How this is scored →
The host as informed peer 4.1 Guest teaching 2.7 Guest disagreement 0.7 The host pushing back 1.2
speaking balance: gold is the host, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 36:39 Deconstructing the 10X Metaphor
Guy rejects the host's tech-centric framing of a '10X prompt engineer', pointing out that creative industries like music production do not measure recording talent using 10X metrics.
Hardest push from the host ▶ 30:41 Refusing the AI as Hobby FramingSteph refuses the framing that generative AI is primarily an entertaining hobby, insisting on its pragmatic integration into e-commerce, scriptwriting, and 3D product printing.
Biggest teaching moment ▶ 7:22 Spatial Dataset LimitationsGuy educates Steph on how image models are trained on archival captions, explaining why they struggle with specific multi-object spatial compositions like describing relative item placements.
The host holds their own ▶ 27:06 Chess Engine Novel Discovery ParallelSteph demonstrates high domain knowledge by comparing generative AI exploration to chess bots surfacing novel moves that human players had missed over thousands of years.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The host as informed peer | Guest teaching | Guest disagreement | The host pushing back | Why |
|---|---|---|---|---|---|---|
| Legal Disclaimer and Program Identification Sequence | 1 | 1 | 0 | 0 | Steph opens the episode with legal disclaimers and introductory context before asking Guy about the origin of his DALL-E prompt book. | |
| Measuring Prompting Experience and Parallels to Other Disciplines | 4 | 2 | 1 | 1 | Steph draws a thoughtful parallel between prompt engineering experience and developer job specs requiring years of experience in newly created programming languages. | |
| Mental Models for Prompting: Searching the Infinite Catalog | 4 | 3 | 0 | 0 | Steph references her prior work at Trends analyzing online datasets, while Guy explains why generative AI models struggle with specific spatial positioning based on image captioning habits. | |
| Alt-Text Training Data, Modifiers, and Prompt Granularity | 5 | 3 | 0 | 0 | Steph demonstrates specific knowledge regarding DALL-E training dataset size and alt-text mechanics, while Guy details image-to-image prompting and selfie app trends. | |
| Navigating the 'Black Box' and AI Stochasticity | 3 | 3 | 1 | 2 | Steph challenges Guy on how to control the generation process rather than repeatedly pulling the AI slot machine until a workable result appears. | |
| AI Model Limitations, Negative Queries, and Composition Hacks | 4 | 4 | 1 | 1 | Steph brings up negative queries and hand rendering glitches, prompting Guy to explain aspect ratio workarounds and Midjourney composition improvements. | |
| Comparing Image Models and Multi-Tool Production Workflows | 4 | 3 | 1 | 2 | Steph frames comparative software analogies and introduces multi-tool production workflows like inpainting and outpainting. | |
| Evolving Interfaces Beyond the Text Bar | 5 | 3 | 0 | 1 | Steph expands on interface evolution by comparing prompt tuning to selling Lightroom filters and suggesting guided onboarding flows. | |
| Specialized Asset Generation and Co-Creative AI Exploration | 6 | 2 | 0 | 0 | Steph demonstrates expertise by introducing the chess engine parallel, where AlphaZero discovered novel strategic moves never considered by humans. | |
| Practical AI Integration and the Broad Spectrum of Design | 4 | 3 | 1 | 3 | Steph pushes back on the idea of AI generation as purely a hobby, insisting on practical commercial applications like 3D printing and e-commerce. | |
| The Career Horizon: 10X Engineers vs. Democratic Tools | 5 | 3 | 2 | 2 | Steph references a Reply All investigative story regarding hyper-specialized music producers to frame the career horizon debate between 10x specialists and mass adoption. | |
| Memes, Cultural Resonance, and Subjective Art | 4 | 2 | 1 | 2 | Steph quizzes Guy on viral imagery, emphasizing that simple memes often outperform elaborate visual art in cultural resonance. |
Statements from this episode (18)
Parsons: Translating visual ideas into language remains a primary design challenge
“It's still quite hard to describe things with words. Designers, yeah, when they go and work, when they do work for clients, like it's one of their pet peeves because clients are like, they don't like it, but they can't explain why.”
Parsons: Art history and design backgrounds give prompt engineers an edge
“So if you've actually been to art school or you're up on your art history or your design language, then you probably got a head start on everyone else.”
Parsons: Prompt AI generators by describing images as existing stock captions
“Like I think if you've never used one before, like the best way to explain how they work is to always like describe something as if it already exists. Imagine that it's an image in some kind of downloadable clip art library, or it's on a photography gallery, a…”
Parsons: AI image models struggle with precise spatial layout prompts
“They often describe generally what the image is about, but not like how you would draw it step by step. And that's why these tools are less good at saying like, I want this thing over here and then that thing next to it and then something on top and that thing…”
Parsons: AI prompts yield diminishing returns as length increases
“And I think there's something to be said, like, I think the longer they are, there's definitely, like, diminishing returns.”
Parsons: Image-to-image AI models will power next-gen consumer interaction
“That's a really interesting space that's gonna probably power like the next generation of how people, especially consumers like interact with these products.”
Parsons: Generative AI fails when forced to produce highly specific work
“That's kind of the limitation of weather technology. Is at the moment, which is, it's amazing until you're trying to do something very specific. And especially if you want to do something very specific, this also to like a very high, like a professional standa…”
Parsons: Midjourney iterates on image models faster than OpenAI
“But then, of course, you have now tools like Midjourney, who've been, like, iterating on their text-to-image model, like, a lot more aggressively than OpenAI, who Understandably, I think maybe have some other things in the cooker, you know, which have now grow…”
Parsons: Stable Diffusion used 12M fine-tuning images for aesthetic quality
“Stable diffusion, I think some five billion and then like a smaller set of like twelve million for the what does nice look like fine tuning that's happened on the top and how they've optimized it.”
Parsons: Generative AI creates a massive market opportunity for editing apps
“Actually, I kind of think it's a big opportunity for, like, the photoshops of this world, because those are tools that presuppose you have some kind of original image to be able to, like, manipulate, whereas now there's a huge amount of, like, raw but maybe no…”
Parsons: Generative AI must evolve beyond text boxes for better usability
“The whole challenge and the whole opportunity, I think at the moment was like, how do you go beyond the text box? How do you go beyond this, like just blank rectangle to create something that is more user friendly, that's more inspiring. That's more how people…”
Smith: Chess AI revealed superior moves humans never considered over centuries
“When we finally built the bots that were better than humans in chess, not only were we surprised by the fact that that could happen, but we were also surprised by all of the different openings or moves that humans in their thousands of years playing chess had …”
Smith: Generative AI value must ladder into practical commercial use cases
“It's fun to play around in these tools, but ultimately there, while there is a market for just interesting art in the world, a lot of this will need to ladder back into, you know, whether it's blog post sharing images, whether it's creating, you know, the next…”
Parsons: Generative AI tools will be used quietly like green screens
“I suspect that when to an extent when you see these things used, especially in prominent contexts, they might not be advertised as such. Much is like green screen, right? Like when green screen is used in films, you shouldn't be like, that is an amazing use of…”
Parsons: AI developers are incentivized to eliminate prompt engineering
“There's obviously every incentive for the people that make these foundational tools to make prompt engineering, for instance, not A thing because they want everyone to be able to do it.”
Parsons: Prompt engineering will be a specialized craft, not universal skill
“I don't think it will become, like, this necessary skill that everyone needs to have, but I do think it will become, you know, like, some people are expert wood whittlers or, you know, really good at Animating hair or whatever, you know, the people that develo…”
Parsons: Careers will emerge around designing hidden AI prompt wrappers
“So there's probably going to be some people whose job is to like, come up with that layer of thing that the consumer or the average person is never seeing. And they think they're just talking to the AI, but really they're talking to this thing that then Adds a…”
Smith: Highly refined AI art does not guarantee audience resonance
“And so your point just reminded me of this idea where art especially Is subjective and what people like and resonate with is not necessarily the most refined or extravagant, precise type of imagery, which you can generate in some of these text to image tools, …”