Shazeer: Centralized AI Inference Batching Is Up to 100x More Efficient
Noam Shazeer · Noam Shazeer: How We Spent $2M to Train a Single AI Model and Grew Character.ai to 20M Users | E1055 · 20VC with Harry Stebbings · Aug 31, 2023 · at 23:46
Noam Shazeer, co-founder and CEO of Character.ai, explains the economies of scale involved in central AI model serving versus running local instances.
0:00 / 0:27exact quote · 27.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Obviously though, there's also like economics of scale of both training the best models And serving, like if you want to serve a product, you can do it maybe a hundred times more efficiently if you're serving many, many people at once and kind of batching things together versus you're, you know, serving like an individual or, you know, or you've got your own rig to run your language models in your basement or something.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Noam Shazeer
Shazeer: Character.ai Rejects Product Managers Who Push for Vertical Specialization
“And like, you know, we talked to like some you know, potential product managers early on and they all say the same thing. Oh yeah. Pick your verticals, narrow it down to me, to make it usable. And like, no, we're not going to hire these people. That's like the…”
Shazeer: Users Prefer Video Game Characters Over AI Psychologists for Therapy
“We put up as an example, like a psychologist character, like maybe, maybe you want to, you know, talk to something and feel better, you know, and you know, that gets a little bit of use, but then what we hear a lot more from users is like, I'm talking to a vid…”
Shazeer: Total Computation Is the Single Most Critical Factor in AI Training
“We can get a lot of, we can get a lot of data, but like, really, actually the number one thing that's important is how much computation you do to train it.”
Shazeer: AI Hardware Advances Will Enable Garage Developers to Match Big Tech
“The hardware is progressing so fast that what you could do at a big company, you know, one year, a few years later, you're going to be able to do You know, at the university lab or in your garage.”
Shazeer: Character.ai Views AI Model Hallucinations as a Feature, Not a Bug
“We consider them a feature or at least, okay, basically, our goal, you know, our strategy is, like, launch something general, let people do what they want with it, and if these models are hallucinating, which we, which they certainly are, and we advertise that…”