Jul 21, 2017 · 44m · y-combinator
Making Music and Art Through Machine Learning - Doug Eck of Magenta · Y Combinator
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Y Combinator podcast episode, host Craig Cannon interviews Doug Eck, lead research scientist for Google's Project Magenta, about leveraging machine learning to build open-source creative tools for musicians and artists. They discuss the technical evolution of generative sound and sequence models, the necessity of building intuitive interfaces for creators, and how AI serves as a collaborative partner rather than a replacement for human artistic catharsis.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The partners as informed peer 2.8 Guest teaching 3.6 Guest disagreement 1.0 The partners pushing back 0.5
speaking balance: gold is the partners, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 26:45 Pushing back on AI alarmism
Doug firmly rejects the hyperbole that generative music is bad for humanity, contrasting moral panic with historical artistic shifts.
Hardest push from the partners ▶ 25:35 Reframing AI writing around pulp fictionCraig challenges Doug's high-concept vision of complex joke generation by arguing AI is far better suited to formulaic airport novels.
Biggest teaching moment ▶ 17:10 The real driver behind deep learning breakthroughsDoug educates on the history of LSTM, dispelling myths by clarifying that modern successes stemmed from compute and scale rather than brand new algorithms.
The partners hold their own ▶ 40:05 Referencing the Gray Area piano demonstrationCraig proves he thoroughly researched Doug's previous lectures, specifically citing his talk contrasting mechanical computer playback with human piano dynamics.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The partners as informed peer | Guest teaching | Guest disagreement | The partners pushing back | Why |
|---|---|---|---|---|---|---|
| Philosophical Framework and Machine Imperfections | 3 | 2 | 1 | 1 | Craig introduces a thoughtful Brian Eno quote on artistic medium failure modes, prompting Doug to reflect on the division between research engineering and artistic breakage. | |
| Audio Generation and Sequence Models in Magenta | 2 | 4 | 1 | 0 | Doug details NSynth and Magenta's architecture, walking through compressed latent spaces and the transition from basic MIDI RNNs to expressive, polyphonic models. | |
| Evaluating Machine Learning Models and User Feedback | 3 | 2 | 1 | 1 | Craig asks how generative quality is evaluated; Doug candidly admits current evaluation is qualitative cherry-picking and jokes about building a viral collaborative filtering app. | |
| Developing Usable Tools for Musicians | 2 | 5 | 1 | 0 | Doug explains why advanced producers like Aphex Twin gravitate toward NSynth's harmonic artifacts and temporal embedding decays rather than typical digital clipping. | |
| Interactive Performance and AI Duet | 3 | 4 | 1 | 1 | Doug clarifies that AI Duet functions as an impulse-response improv partner for skilled jazz musicians rather than an autonomous long-form composer. | |
| History and Evolution of Recurrent Neural Networks | 3 | 5 | 0 | 0 | Doug recounts working at IDSIA with Schmidhuber and Alex Graves, explaining how RNN breakthroughs were fundamentally enabled by scaled compute and massive data rather than algorithmic magic. | |
| Structural Challenges in AI Music and Caricature Generation | 3 | 4 | 1 | 1 | The conversation covers model limitations in handling long-term musical hierarchy, noting how low-temperature sampling yields caricatures like the platonic Bach or average cat. | |
| Future Horizons: Narrative, Jokes, and High-Level Structure | 3 | 3 | 1 | 1 | Doug explores high-dimensional narrative and joke construction, while Craig counters by suggesting generative models might first excel at formulaic airport pulp fiction. | |
| Public Reaction and Positioning AI as Creative Tools | 2 | 3 | 2 | 1 | Doug dismisses alarmist claims that generative models harm humanity, framing AI systems as expressive artist tools rather than push-button automated replacements. | |
| Catharsis in Art Creation and Code Quality | 3 | 3 | 1 | 0 | Doug discusses the emotional catharsis of creative coding and training models, comparing it to programming drum machines rather than merely playing preset loops. | |
| Reinforcement Learning and Rule-Based Steering | 3 | 5 | 1 | 0 | Doug explains how deep Q-learning and reinforcement learning can steer pre-trained generative models with rule-based heuristic evaluators like counterpoint or line curvature. | |
| AI Impact on Pop Music and Cultural Adaptation | 3 | 3 | 2 | 1 | Doug demystifies the fear of AI creating the single perfect pop song, arguing that cultural scaffolding shifts artists toward tackling new, harder forms of human expression. | |
| Long-Form Composition and Expressive Musical Timing | 3 | 4 | 0 | 0 | Craig highlights Doug's earlier piano demonstration, leading into a discussion of Thelonious Monk's expressive timing and the need for fluid real-time MIDI APIs. |
Statements from this episode (22)
Eck: Magenta aims to build creative tools, not act as artists
“The reason that I put that quote there, I think, is to be honest with the division between engineering and research and artistry, and to not think that what I'm doing is being a machine learning artist, but we're trying to build interesting ways to make new ki…”
Eck: Users' first instinct with AI art models is breaking them
“The first thing you're gonna do, if you think, if someone comes to you and says, here's this really smart model that you can make art with, what are you gonna do? You're gonna try to show the world that it's a stupid model, right? But maybe the way that, maybe…”
Eck: Google Magenta's NSynth cannot synthesize audio in real time yet
“Right now, it's quite slow to listen, so to speak. We're not able to do things in real time.”
Eck: Google Magenta's early music generation models were primitive RNNs
“We put out some models that were, you know, by any reasonable account primitive, I mean, kind of very simple recurrent neural networks that that generate MIDI from MIDI”
Eck: Magenta's generated AI music is not yet good enough for testing
“I, at least for Magenta, I haven't felt like the quality of what we've been generating has been good enough to bother, so to speak. Like you find it, you cherry pick, you find some good things, you're like, okay, this model trains, and it's interesting, and no…”
Eck: Artists will not do a huge amount with Quick Draw data
“I think, you know, I'm not expecting that artists really do a huge amount with this quick draw data because as cool as it is, These things were drawn in 20 seconds, right? There's kind of a limit to how much we can do with them.”
Eck: Audio model generation errors sound musical because of learned training variance
“I think they're interesting because the model has been forced to capture some of the important sources of variance in real music audio, and even when it Fails to reproduce all of them when it fills in with confusion, so to speak. Even that confusion is, is som…”
Eck: Google AI Duet relies on simple 2002 RNN technology
“And what I noticed, even with AI Duet, which is this like web-based, like, it's a simple RNN. It's like, I can lay claim. It's technology that was published in 2002. It's really a very simple, really simple.”
Eck: AI music tools require strong human improvisation skills
“But if you don't have the musical, so basically it's the musician bringing all the skills to the table. Right. So like, even, even with the primitive sequence generation stuff, like it's still been interesting to see that it's musicians with a lot of musical t…”
Eck: Magenta prioritized timbre generation over sequence generation
“I haven't, well, we haven't pushed the sequence generation stuff much because we really wanted to focus on, on tamper.”
Eck: Alex Graves advanced LSTMs more than anyone, including its creator
“Among the three of us, by far, Alex Graves has done the most with LSTM. So he continued, after he finished his PhD, and he continued doggedly to try to understand how recurrent neural networks worked, how to train them, and how to make them useful for sequence…”
Doug Eck: Vanilla LSTMs cannot handle long-timescale hierarchical musical patterns
“Like LSTM in its most vanilla form, I think everybody's pretty convinced that it's not going to handle really long time scale hierarchical patterning.”
Eck: Untrained listeners rated Magenta compositions as more Bach-like than Bach
“When we put these tunes out for, like, untrained listeners to listen to, they sometimes voted them as sounding more Bach-y”
Eck: Low-temperature AI models produce caricatures rather than authentic samples
“If you sample from Sketch RNN with very low temperature, meaning with, like, without a lot of noise in the system, you actually get what, like, if you want to squint your eyes and break philosophy is like the platonic cat. You know, you get a cat that looks mo…”
Eck: As of 2017, AI cannot generate a single coherent text paragraph
“So, so everybody understands that's listening or watching, you know, we can, we can't generate a coherent paragraph, right? So, I don't mean we, magenta. I mean, kind of we, humanity.”
Eck: RL is slower to train than GANs but offers more flexibility
“So another way to do this is to use reinforcement learning. Yeah. And it, it's slower to train, because all you have is a single number, scalar reward, instead of this whole gradient flowing. Than GANs, but it also is more flexible.”
Eck: Adding Deep Q-learning rules made Google Magenta's music noticeably catchier
“And we had, I thought, some very nice generated samples of music that were pretty boring with the LSTM network. But then the LSTM network trained additionally using a kind of reinforcement learning called deep Q learning to follow some of these rules. The gene…”
Eck: AI will automate routine music creation, pushing artists toward new complexity
“Like some things that used to be hard will be easy. And so, We'll offload all of that. And if people are happy just listening to the stuff that's now easy, then yeah, it's a problem solved and we'll be able to generate lots of it. But then what people tend to …”
Eck: Automating rhythm with drum machines enabled more complex human vocal phrasing
“So you've solved the metronomic, you know, beat problem. And then what you actually find is that artists who are really good at this, they play off of it and they're allowed like when they sing to do many more like rhythmical things than they could do before. …”
Eck: Magenta aims to generate coherent long-form music and art
“I think composing, creating long form pieces, whether they're music or art, I think is, is something we want to do. And this hints at this idea of like, not just having these things that make sense at like, 20 seconds of music time, but actually say something …”
Eck: Long-form ML models will let composers delegate macro structure
“If we get to models that can handle longer structure and nested structure, we'll have a lot more ways in which we can decide what we want to work on versus what we have the machine help us with, right?”
Eck: Magenta has a core API enabling real-time MIDI AI collaboration
“I think the API, the core API that allows us to move music around in real time in MIDI, and actually have a meaningful conversation between an AI model and multiple musicians is there, and there's just a bunch more thinking that needs to happen, right, to get …”