The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 21 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Biewald: Most enterprises have not deployed LLMs into production yet
“I think that LLMs in particular, we talk to a lot of the people and we don't see a ton of people getting them into production yet. And I think it's funny, like VCs are always surprised, like when we tell them that I think that I don't know. I'm bullish on LMS,…”
Biewald estimates 99% of Global 2000 use ML for core operations
“I bet 99% of the global 2000 is using machine learning for something that they actually really care about.”
Biewald: PyTorch beat TensorFlow through developer empathy, not eager execution
“I don't think they really, I think people tell this, the story of sort of the silver bullet. Of like you know, the eager execution model. But I think the reality is they just built a product with so much more empathy.”
Biewald: Almost all major LLMs were trained using Weights & Biases
“I think all of the major LLMs out there, almost all were trained using weights and biases.”
Biewald: Simple operational errors cause more model failures than data drift
“People talk a lot about data drift in the industry. And that's this idea that like, you know, like language changes over time and you want to know that it's changing and sort of like have your model you know, notice that and update it. But I guess like what I …”
Biewald: OpenAI is a W&B customer with a small number of production models
“OpenAI has been, like, a longtime customer. I mean, I consider them, like, extraordinarily sophisticated, and they have a pretty small number of models in, in production, so.”
Increasing dataset accuracy from 90% to 95% repeatedly halves error rates
“So this is my same data set that I published online, but if you take it from 90% to 95%, you actually have the error rate, and then going up to a hundred percent, you have it again, right?”
Biewald: Data labeling software requires a top-down sales model
“Data labeling, I think really wants to be a top down sale”
Biewald: LLM API developers are less mathematically specialized than traditional ML engineers
“Even the people Working with a lot of these APIs you know, are, like, less huge math nerds than, you know, some of the people that have been training, you know, models for a long time.”
Biewald: Developers building with LLMs focus heavily on qualitative anecdotes over metrics
“In the LL world, like the anecdote is something people really pay attention to.”
Biewald: Traditional ML methods like boosted trees remain a large share of W&B usage
“A lot of people still running, you know, boosted trees or, you know, random forests inside of weights and biases. So we, you know, it's like actually huge. We should probably do a block, but it's still a big fraction of our You know, of our user base.”
Biewald: Pharma is investing far more in deep learning than realized
“I think pharma is investing way more in deep learning than people realize.”
Biewald: Technical buyers don't want sales dinners, they just want facts
“Nobody wants to golf or anything. I mean, that's for sure, right? Like, I mean, there's people like a super aggressive salesperson. Some of my salespeople are really competitive. I am actually really competitive myself, but they kind of like suppress it in a w…”
Biewald: Only five out of 100 feature engineering attempts actually improve models
“Every task that you work on has kind of different different features work better or worse, and I spent, when I first was working as a data scientist, I spent all my time on this feature selection, and as you guys know, you try a hundred things and maybe five o…”
CrowdFlower pays all global crowd workers the same flat compensation rate
“We pay everyone the same regardless of what country you come in from.”
Biewald: No single industry accounts for over 15% of W&B revenue
“No one vertical is more than like, you know, 15% of our usage or revenue or anything like that.”
CrowdFlower charges a platform fee plus up to a 20% take rate
“We have a platform fee to use our software, and then we take a, up to a 20% cut of the work that, that flows through our platform.”
CrowdFlower spent just $100 to collect 10,000 initial color labels
“We collected that data. It maybe cost us a hundred bucks to get 10,000 labels.”