Fireworks AI serves fine-tuned LoRA adapters at base model pricing
Lin Qiao · Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI · Nov 25, 2024 · at 52:11
Lin Qiao, CEO of Fireworks AI, explains multi-LoRA serving architecture and pricing on their platform.
0:00 / 0:25exact quote · 25.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We wrote multi LoRa last year, actually, and we actually have this function for a long time and many people have been using it, but it's not well known that, oh, if you find your model, you don't need to use on demand. If you find your model is LoRa. You can upload your LAR adapter and redeploy it as if it's a new model. And then you use, you get your endpoint and you can use that directly. But at the same cost as the base model.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Lin Qiao
Specialized open-source expert models will outperform one-size-fits-all closed-source models
“And that's our prediction is With specialization, there will be a lot of expert models, really, really good, and even better than, like, one size fits all open source closed source model.”
AI inference matters more than training because it scales with global population
“Our prediction is for those kind of applications, the inference is much more important than training. Because inference scale is proportional to the upliminal world population. And training. Training scale is proportional to the number of researchers.”
Compelling GenAI applications require compound AI systems spanning multiple modalities
“In order to really build a compelling application on top of JNI, we need a compound AI system. Compact AI system basically is going to have multiple models across modalities along with APIs, whether it's public APIs, internal proprietary APIs, storage systems,…”
Fireworks AI will release a reasoning model inspired by OpenAI's o1
“So another announcement is we will also announce a, our next
Declarative system is going to be appear as a model that has extremely high quality, and this model is inspired by O-one announcement from OpenAI.
You should see that by the time we announce this o…”
Pre-training on human data is hitting limits; synthetic data is required
“So I think on the data side, we're approaching the limit and the only data to increase that is synthetic generated data.”
Fireworks AI's multi-LoRA system serves up to 1,000 adapters per base model
“One base model can sustain a hundred to a thousand LoRa adapters. And then basically all these different LoRa adapters can share the same, like direct the same traffic to the same base model where base model is dominating the cost.”