Assertion Partly supported AI assessment confidence: 85% certainty 4/5 debate potential 2/5

Firshman: Llama 2 costs $25M to train but $50 to fine-tune

Ben Firshman · A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate · Feb 28, 2024 · at 1:14:10

Ben Firshman, CEO of Replicate, contrasts the massive pre-training costs of foundation models with the negligible cost of fine-tuning.

0:00 / 0:06exact quote · 6.8s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“Lama II as a base model is that, like, yeah, it costs twenty five million dollars to train to start with, but then you can fine tune it for, like, 50 bucks.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

A base model like Meta's Llama 2 costs approximately twenty-five million dollars to train initially, but can be fine-tuned for around 50 dollars.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

The claim bundles two separate cost assertions: the cost to train the initial base model and the cost to fine-tune it. The second assertion holds: standard parameter-efficient fine-tuning (e.g., LoRA/QLoRA) on cloud GPU instances (such as AWS EC2 g5 instances or RunPod) or managed platforms (such as Replicate or Together AI) consistently cost between $5 and $50 for Llama 2 models. However, the first assertion is materially inaccurate regarding base model training: Meta's technical disclosures (July 2023) reported 3,311,616 A100 GPU-hours for pretraining the Llama 2 suite, which independent benchmarks and the Stanford AI Index / Epoch AI (2024) priced at approximately $3.9M to $5.0M in compute, not $25M (a figure that analysts such as Nathan Lambert attributed specifically to the separate expense of human RLHF data annotation). Because one leg holds and the other materially overstates base pretraining compute costs, the claim is partially true.

Meta's July 2023 technical report disclosed total Llama 2 pretraining compute at 3,311,616 A100-80GB GPU hours.
The Stanford AI Index Report 2024 and Epoch AI estimated the primary compute cost of Llama 2 70B at approximately $3.93M, with the full suite estimated at $4.0M to $5.0M.
Analysts (e.g., Nathan Lambert, July 2023) estimated human data collection and RLHF annotations at ~$25M, rather than base pretraining compute.
Commercial cloud GPU platforms and LoRA/QLoRA fine-tuning workflows routinely enabled Llama 2 fine-tuning for $5 to $50.

Checked against

[1] huggingface.co [2] voronoiapp.com [3] visualcapitalist.com [4] arxiv.org [5] interconnects.ai [6] latent.space [7] youtube.com [8] medium.com

"Replicate" "fine-tuning" "cost" "llama 2" "25 million" "Llama 2" train OR training OR cost "Epoch AI" "Llama 2" training cost "Llama 2" "3.3M GPU hours" OR "3.3 million GPU hours" cost "Llama 2" "GPU hours" training compute Touvron "Llama 2" fine-tuning cost dollars LoRA cost to train llama 2 estimate 25 million "Replicate" OR "Anyscale" OR "Together AI" "fine-tuning" "Llama 2" "$50" OR "50 dollars"

assessed as of 2026-09-03

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Facebook Llama 2

⚑ Spot an error? Report it

More from Ben Firshman

Insight

Firshman: Academic ML researchers make poor software users due to six-month cycles

“And they were like, what was funny about that is they were like, not very good users. Like, they were doing great work, obviously, but the way that research worked is that they just made, like, one thing every six months, and they just fired and forgot it. Lik…”

Ben Firshman Feb 28, 2024 ▶ 28:01 A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Assertion Contradicted

Firshman: Early 2021 Discord AI bots originated Midjourney's collaborative interface

“It was the start of, it was the start of mid-journey, and, you know, it's where that kind of user interface came from. Like, what's beautiful about the user interface is, like, You could see what other people are doing, and that you could riff off other people…”

Ben Firshman Feb 28, 2024 ▶ 31:20 A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Assertion Not checkable as stated

Firshman: GPU demand is not currently outpacing supply at Replicate

“From our point of view, demand is not outpacing supply of GPUs. Like we have enough, from our point of view, we have enough GPUs to go around, but that might change for sure.”

Ben Firshman Feb 28, 2024 ▶ 1:05:09 A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Opinion

Firshman: Building Fig and Docker Compose in Python was a mistake

“We used Python, which was a big mistake, where Python's really hard to get booting up fast, because you have to load up the whole Python runtime before it can run anything.”

Ben Firshman Feb 28, 2024 ▶ 2:54 A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Insight

Firshman: CLIs are a more natural fit for LLMs than GUIs

“It's almost more natural to a CLI than it is in a graphical user interface, because it feels like there's back and forth with the computer. Yeah. Almost funnily like a language model. So I think there's some interesting intersection of like CLIs and language m…”

Ben Firshman Feb 28, 2024 ▶ 6:26 A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Disclosure

Firshman: Llama 2 release was Replicate's biggest week of growth ever

“Llama II was, like, our biggest week of growth ever, because, like, tons of people wanted to tinker with it and run it.”

Ben Firshman Feb 28, 2024 ▶ 39:20 A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Firshman: Llama 2 costs $25M to train but $50 to fine-tune

More from Ben Firshman

Turn any episode into a week of clips.