transformer architecture
also referred to as: transformer architectures
5 statements across 5 episodes · 2 bullish · 1 bearish · 5 people on the record · first statement Nov 11, 2024 by Stanislas Polu · across every show →
Everything said about transformer architecture, oldest first
Nov 11, 2024
Mar 13, 2025 neutral
Shankar: LLM failure modes and evaluation techniques have largely stabilized
“I think techniques have stabilized. I think the kinds of failure modes of LLMs, I mean, they're still there, but it's not like changing every single day. We know that LLMs are bad at certain things. We know a little bit more about say limitations of the transf…”
Oct 1, 2025 positive
Jun 18, 2026 positive
Midha: Anthropic's velocity came from standardizing on the transformer architecture
“Like, one of the reasons Anthropic has had extraordinary sort of velocity is because they picked the transform architecture and said, this is simple, let's double down on it, right? And now, luckily, there's enough investment going into space that we can affor…”
Sep 4, 2026 negative
Anandkumar: Standard Transformers cannot scale to 5-trillion context lengths for physics
“On the other hand, if you think about using transformer architectures that have worked so well for language, that just wouldn't be able to support a five trillion context length. No matter all the compute in the world is thrown at it. So that kind of quadratic…”