Mar 14, 2024 · 58m · latent-space
Making Transformers Sing - with Mikey Shulman of Suno
Mikey Shulman · 32m spoken Alessio Fanelli · 7m spoken Shawn Wang · 6m spoken Generated Vocals · 4m spoken Generated Audio · 3s spoken
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Live in Space podcast, co-hosts Alessio Fanelli and Shawn Wang interview Suno co-founder Mikey Shulman about the technical architecture, product design, and creative vision behind their state-of-the-art music generation platform. Shulman provides live v3 model demonstrations while discussing audio tokenization, consumer-focused workflows, and the future of participatory music creation.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 14% of the talking time here. How this is scored →
The hosts as informed peer 5.2 Guest teaching 5.1 Guest disagreement 1.1 The hosts pushing back 1.1
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 40:56 Pushing back against AI novelty covers
Mikey firmly rejects the popular AI music trend of voice cloning and novelty covers, comparing them to disposable ChatGPT sonnets and explaining Suno's active restrictions on copyrighted lyrics.
Hardest push from the hosts ▶ 14:15 Challenging Suno's music-only positioningSwix playfully challenges Mikey's assertion that Suno is exclusively a music company by citing their recent speech model release with NVIDIA.
Biggest teaching moment ▶ 8:30 Non-musical vocal datasets for singing synthesisMikey educates the hosts on multimodal training recipes, explaining that realistic singing voice generation requires incorporating non-musical human vocal data into the training corpus.
The host holds their own ▶ 54:24 Framing ML evaluation through Goodhart's LawAlessio demonstrates deep familiarity with Mikey's technical writing by bringing up his Kensho blog post on Goodhart's Law to explore LLM benchmark limitations.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Introductions and Mikey Shulman's Background | 5 | 5 | 1 | 1 | Swix and Alessio open with Mikey's background and ask how music generation works compared to text LLMs and image diffusion. Mikey explains that audio lags text/vision by two years and details how transformer autoregression operates on discrete audio tokens. | |
| Training Data Recipes and Multimodal Audio Learning | 6 | 6 | 1 | 1 | Alessio asks about the training dataset recipe and whether high-quality music follows power laws similar to text LLM datasets. Mikey reveals that Suno trains models on non-musical human vocal audio alongside music to improve vocal synthesis realism. | |
| Scaling Audio Models and Real-Time Latency Constraints | 5 | 5 | 1 | 1 | Alessio queries model parameters and local execution feasibility. Mikey discusses real-time streaming latency constraints and explains why relying purely on massive scale can be a research crutch that discourages model optimization. | |
| The Origins of Suno and Choosing Music Over Speech | 4 | 4 | 1 | 2 | Swix asks why Mikey chose music over more commercially conventional speech applications, playfully pointing out recent speech collaborations. Mikey explains that emotional resonance and fun drove their organic focus towards music. | |
| Open-Source Foundations of Bark and Latent Music Behavior | 6 | 5 | 2 | 1 | Swix probes Bark's lineage, initially mischaracterizing it as speech recognition before Mikey clarifies it is text-to-speech built on nanoGPT concepts. Mikey explains how self-supervised training allowed latent musical behavior to emerge naturally. | |
| Suno User Modes and Social Dynamics of Music Sharing | 5 | 5 | 1 | 1 | Alessio asks about casual vs. power user behavior on Suno. Mikey reveals over half of users engage with expert mode for fine-grained lyrical tweaking and discusses how micro-sharing music brings joy to small social circles. | |
| Consumer Workflows Versus Professional Music Production | 6 | 5 | 1 | 2 | Alessio references Madlib's iPad production to question if Suno aims to capture professional DAW workflows. Mikey clarifies that Suno deliberately targets consumer participation rather than incremental productivity for professional audio engineers. | |
| Live Demo: Generating GPU Cloud Blues and Country Songs | 4 | 4 | 1 | 1 | Alessio prompts Suno to generate a country song about missing cloud GPUs. The hosts and Mikey analyze the rapid latency, vocal generation, and domain knowledge reflected in the generated output. | |
| Live Demo: Generating House Music and Prompt Tuning | 5 | 4 | 1 | 1 | Swix requests a house music track about podcasting, prompting Mikey to demonstrate real-time prompt modification. Swix notices special tokens like beat drops and asks if they are training artifacts, which Mikey clarifies are emergent user prompt strategies. | |
| Live Demo: Style Modulation and Apple Vision Pro Blues | 5 | 5 | 1 | 1 | The hosts test model boundaries with Apple Vision Pro themes and blues styles. Mikey discusses guardrails against impersonation when location names like Chicago trigger filters and explains why repeating prompt keywords too often degrades output. | |
| Copyright Policy, Novelty Covers, and Original Music | 6 | 6 | 2 | 1 | Alessio brings up sample flipping and remixing classic tracks in hip-hop. Mikey emphasizes that Suno intentionally blocks copyrighted lyric reuse to steer users toward creating original music rather than viral novelty covers. | |
| Future Roadmap: Collaborative Concerts and Active Music | 5 | 5 | 1 | 1 | Mikey contrasts the 50x larger gaming industry with music's passive consumption model. Swix suggests a Twitch Plays Pokemon radio stream while Mikey shares his vision for interactive, audience-driven collaborative concerts. | |
| Model Personalization, Feedback Loops, and User Impact | 5 | 5 | 1 | 1 | Swix asks how Suno handles subjective audio quality feedback loops and artifacts. Mikey details plans for personalized producer models and highlights Suno's adoption within visually impaired communities. | |
| The Generative Audio Landscape and Music Production Tools | 5 | 6 | 1 | 1 | Swix asks Mikey to map out the generative audio ecosystem. Mikey categorizes the landscape into royalty-free stock music, AI covers, consumer original generation, and professional DAW plugins/stem splitters. | |
| Goodhart's Law, Aesthetic Evaluation, and Conclusion | 6 | 6 | 1 | 1 | Alessio references Mikey's writing on Goodhart's Law. Mikey explains why quantitative ML benchmarks fall short in audio evaluation and why social scientists and economists make great ML engineers by thinking from first principles. |
Statements from this episode (23)
Shulman: Generative audio AI lags text and images by one to two years
“So I think very roughly you can think audio is like one to two years behind images and text. And so you kind of have to think today like text was in 20, 22 or something like this.”
Suno avoids hardcoding musical rules into its generative models
“We try not to impose anything about music or audio in general into the model, and we kind of let the models learn things by themselves.”
Shulman: Suno focuses primarily on audio tokenization to leverage text transformers
“What we do is we benefit from all of the beautiful things people do with transformers and text, and we focus very hard basically on how do I tokenize audio in the right way.”
Shulman: Understanding of music scaling laws is far behind text
“I don't think we know these things nearly as well as they're known in text. We have some notions of some of the scaling laws here, but I think yeah, we're just so, so far behind.”
Shulman: Suno does not only train its models on music
“People are always surprised to learn that we don't only train on music.”
Shulman: Suno is particularly bad at capturing realistic vocals
“One of the places that we are particularly bad is vocals and at capturing really realistic vocals.”
Streaming throughput constraints prevent Suno from scaling to 175 billion parameters
“We care a lot about how many tokens per second we can generate because we need to stream new music as fast as you can listen to it. And so that is a big one that I think probably has us never get to a hundred seventy-five billion parameter model, if I'm being …”
Shulman: Suno's first release was open-source TTS model Bark
“In fact, we, the first thing we ever put out was a speech model. It was Bark. It was this open source text-to-speech model, and it got a lot of stars on GitHub,”
Shulman: Bark was the first open-source transformer-based TTS model
“As far as I know there was no other certainly not in the open source text to speech that was kind of transformer based.”
Shulman: Suno borrowed significant code from Andrej Karpathy's nanoGPT for Bark
“There's a big shout out to Andre Karpathy's nano GPT. You know, there's a lot of code borrowed from there.”
Shulman: Suno aims to undo cultural barriers preventing people from making music
“There's actually a lot of cultural forces that kind of cue you to not think to make music and that's kind of what we're trying to undo.”
Shulman: Over half of Suno usage is in expert mode
“Yeah, actually more than half of the usage is that expert mode.”
Shulman: Accessible AI music will drive private small-group sharing
“But when you start to make that more accessible to people, they are going to share music in much smaller groups, maybe even not at all, but like with one person or three people or five people.”
Fanelli: Madlib produced full Freddie Gibbs album entirely on iPad
“Madlib actually produced this whole album with him and Freddie Gibbs produced the whole thing on an iPad. He never used a computer.”
Shulman: Professional musicians are using Suno for inspiration and sample generation
“There are lots of professionals that we know about using our stuff, whether it's for inspiration or sample generation and stuff like that.”
Shulman: Suno restricts prompts to prevent artist impersonation
“We try to be very careful not letting you impersonate, and it is possible.”
Shulman: Repeating prompt keywords too often degrades music model outputs
“It's actually, you can't really repeat too many times. You kind of, it gets like the hypothesis gets like a little too out of domain.”
Shulman: Suno prohibits users from generating songs using copyrighted lyrics
“We actually don't let you do that. And it's because if you're taking someone else's lyrics, you didn't own those. You don't have the publishing rights to those. You can't remake that song.”
Shulman: AI song covers are viral novelties, not music's future
“And I think this stuff is very viral, but I actually really don't think that this is how people want to interact with music in the future.”
Shulman: Gaming dwarfs music 50x because music is mostly passive consumption
“And, you know, as a very broad heuristic, the gaming industry is 50 times bigger than the music industry. And it's because gaming is super active. And music, too much music is just passive consumption.”
Suno is seeing significant adoption among blind and vision-impaired users
“We're fairly popular in the blind and vision impaired community”