AI21 Labs

“AI systems need to be model agnostic. They shouldn't care about which model that they use, and they should look at what I call actions, which is a combination of a model with a prompt and maybe a set of tools that it can use and say, what can an action do for …”

“And we're using our own infrastructure to train our models. There still isn't an open source infrastructure that I could say, use this to train your very, very large model.”

“Since then, we've released several models, recent model lines in called Jamba, which I think the fascinating part about it is, is the first hybrid model. It's not just attention.”

“And I'm looking for people that try to solve problems on their own. Because if you think you're gonna found the answers online, I think you're not in the frontier.”

“Putting it the first or the last performed worse than in the middle. And one to eight was good enough. You know, you might get very slight improvements with one to six, but it was marginal, maybe within the standard deviation.”

“I can definitely see sequence length rising, and I can't see full attention models being as prominent as they are today. So, so, so at least they'll have less full attention layers and I hope they'll have more innovations like Mumbai.”

“If I had to guess hybrid models are here to stay just because the efficiency without sacrificing the performance is, is too much to give up. You know, the attention is so expensive. The quadratic cost and the linear memory cost is so much that I think for long…”

“And we're actually releasing new versions of our Jamba models soon, including a three B dense model. It's aimed for long context on edge devices.”