Albrecht: CARBS models compute cost per sample for hyperparameter search
Josh Albrecht · State of the Art: Training 70B LLMs on 10,000 H100 clusters · Jun 25, 2024 · at 47:25
Josh Albrecht, CTO of Imbue, explains how CARBS differs from traditional Bayesian hyperparameter optimization frameworks.
0:00 / 1:10
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“CARBS is, it's maybe a backronym, but it's for Cost Aware Pareto Region Bayesian Search... The point is that it's a cost aware hyperparameter tuner. So most hyperparameter tuners you kind of say, okay, here's this objective function. I want you to make this number as big as possible or as small as possible... That's usually saying like, you know, almost all of these hyperparameter configurations are, let's say they're all going to use the same number of GPUs or the same number of nodes. So it's going to run for the same amount of time... But what carbs does is it says, okay, actually, what if we relax that constraint? What if we say each of these different points, we're going to model how expensive it will be to sample this configuration.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Josh Albrecht
Albrecht: LLM emergence is an artifact of non-linear evaluation metrics
“This emergent behavior that you're seeing, Is not really emergent behavior, but is really a function of the evaluation metrics that we're using.”
Albrecht: Training on AWS prevents diagnosing low-level hardware errors
“And if we're just using, you know, AWS or some other cloud provider, These errors are still going to be there, and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong.”
Albrecht: Benchmark performance differences vanish once ambiguous questions are cleaned
“The main takeaway from any of the, like, actual performance is like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's important to look at like, you know, like when you're talking about performance on NL…”
Albrecht: AgentBench paper's appendix examples are actually incorrect solutions
“Like we were looking at the agent bench paper, I think just last week for our paper club. And one of the things that we noticed is that actually like both of the examples in the appendix that are given as like traces where it got it right. This is actually not…”
Albrecht: Optimizing for competitive coding benchmarks does not create useful programmers
“Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones, so that you can go very far down that route and make something like really good at those problems, but not actually that useful…”
Albrecht: Imbue avoids Kubernetes to keep cluster infrastructure simple to debug
“Less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, for example, I would just directly launch these things and it's just been much easier to debug this way.”