MLP

“And I would say the original sin of Transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and …”

“So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to like, use its knowledge to think and then give me a second answer, or like, should I hir…”

“Yeah, on Sudoku, MLP actually outperformed the attention, It was it scored zero on the maze, ah, the MLP scored zero on the maze, and so there's, it's not clear, it's not obvious that, ah, the transformer is always better.”

“Transformer does everything. Mamba does most of it. LSTMs do only partially, and MLPs fail completely.”

“These are just MLPs. So the model has both attention heads and multi-layer perceptions MLPs. We don't just do it. Like we completely ignore attention or like we don't try to decompose it at all. So there's some prompts where like all of the interesting stuff i…”

“So like, I think that right now we have some pretty good solutions for like understanding what's in the residual stream, understanding what's, is it in MLPs? We don't have good solutions for like attention.”

“We sold a number of packages early on in the investment to MLPs, to these production-based partnerships that were public trading at good values, and we were able to lock in a lot of capital to cycle into land over time, and that's how we kept the initial inves…”

“So, 10 years ago, there was a private public ARB in the MLP space. You could cobble together the MLP Greenfield at three, four times EBITDA, and then you could sell it to the public markets at 12 or 14 times EBITDA.”