The Ledger, every show

“What we're able to see is that, you know, you can search ESMC and you can actually find antibodies that are reaching the level of affinity that are, I should say, are really at the level of affinity that is needed for therapeutic function and activity.”

“Yeah, I mean, I think we're state of the art for open models.”

“We're gonna have increasingly capable and accurate digital representations of molecules, genomes, cells, ultimately physiology. That's where you want to get. We're gonna have to go up that, that complexity scale, the levels of biological complexity that requir…”

“One of the classes of models that we train are these protein language models. So they're really, you know, it's trained on the codes of proteins, and so anything they learn about biology is, is kind of emergent. And we've seen that they can learn things like b…”

“And it's really, you know, hitting state of the art across pretty much every structure prediction benchmark, and especially on protein, protein interactions and protein antibody interactions, which is really critical for therapeutic design.”

“So our team has really explored that idea over a number of different years, and we've really kind of, I think, seen the scaling curve and really seen as we have increased models by an order of magnitude kind of in each generation that, you know, there's this e…”

“You know, what we found basically is that the model has a kind of a single feature for this nucleophilic elbow and is activating across these like very evolutionarily diverse families, you know, really completely different structural topologies, proteins that …”

“And then, you know, what we saw basically is, is, is there are no longer diminishing returns to scale. So that's really saying that ESM two was kind of data limited rather than compute limited for ESMC.”

“We've been able to use this to actually now go and design many protein binders, but I think sort of most excitingly, we've been able to use this to actually design antibodies, SCFVs, and we're seeing really, I think, exciting success rates and a small number o…”

“The hope is that you kind of really learn the underlying basis for how it's making the predictions, and so you open up the black box and you can actually understand kind of the biology that the model is representing.”

“So we folded over 1.1 billion proteins and predicted their structures and identified kind of features connecting all of them through mechanistic interpretability.”

“And so my team, when we were at MetaFair, trained really the first transformer language model for protein biology.”

“So we put together kind of all the world's largest protein sequence databases. And so that kind of amounts to 6.8 billion non-redundant proteins, and then we've resolved predicted structures for 1.1 billion of those, and we've also computed features across all…”

“ESM-II is trained on Uniref. And for ESMC, we added metagenomics. So we added billions more sequences to the training data.”

“Actually, the first version of the ESM atlas was used by Fang Zhang's group to find A new gene editing system.”

“So the other thing about ESM fold two is a really fast model because it doesn't require the multiple sequence alignment. So You know, you can do inference kind of, you know, directly from the sequence it takes seconds, you know, you can get an atomic resolutio…”

“I think with current technology, you know, we can definitely kind of get, get data, 10 X to a hundred X where it is today with like relatively reasonable investments, you know, but then to get another 10 X or more in there, that's going to require, require a l…”

“I think one of the really interesting things that we've been seeing is people kind of connecting it with agentic systems to just kind of do automated design and kind of just automate that, that whole process.”

“So there's actually three models in that family. There's a three hundred million parameter model, a six hundred million. Parameter model and a six billion parameter model.”