Transformer Models
topic on 8 shows · 13 statements across 13 episodes
Acquired 
Cheeky Pint 
Latent Space 
the Neon Show 
No Priors 
the MAD Podcast 
Big Technology 
20VC
13 statements about Transformer Models, every show
Kutylowski: Task-specific AI models outperform general LLMs due to dedicated parameter capacity
“When they're made for different purposes, they also lose a little bit of the capability that they had maybe initially when they've been made for translation only. This The set of parameters that is available there, this, which kind of determines quite often th…”
Staniszewski: ElevenLabs applied transformer and diffusion concepts to voice
“And here credit to my co-founder, Piotr, who effectively came with that new idea of how you can now create voice models, which are both reliable, high quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models into …”
Collison: Everything will pass through a transformer model before consumption
“Everything will pass through a transformer model before it's consumed.”
Namasivayam: Transformer models miss edge-case opinions by defaulting to population averages
“Transformer models, OpenAI, is great at providing feedback on the average, but it doesn't cover the edge cases. Or the edge cases of your opinions. So that's the problem. So ChatGPT is good at persona feedback. It kind of averages it across the population. But…”
Jamil: 'Writing in the Margins' works on any transformer without fine-tuning
“So it can be used with any transformer model without fine-tuning, just by doing it, just by doing this inference differently.”
Moody's automated feature engineering using transformer models on text data
“Then we actually automated the feature engineering process, for example, using transformer models to extract features from massive, massive text data.”
Ramaswamy: Google open-sourced Transformers in 2017 to prevent top researchers from leaving
“If you had told these people at that time that they could not publish, they would have gone and worked for universities, they would have gone and worked for Microsoft, they would have gone and worked for other people.”
Scaling compute and data created unpredicted emergent reasoning in language models
“We don't change anything about the structure, we just give it way more data and let it Run these models for a long time and make the parameters of the model way bigger, and like, no researchers expected them to reason about the world as well as they do, but it…”
Tobi Lütke: Tech leaders must understand AI models, not use black boxes
“I need to not treat transformer models as a black box. Like I need to understand those because otherwise I cannot show up to a meeting with my teams and have a good chance of reasoning with them about not just what is sort of a convenient next task, but what's…”
Liang: Language models should use calculators instead of computing internally
“There are cases where you want to just map natural language into say people call it tool use. Like you ask some question that reverse calculation, you should just use a calculator rather than trying to sort of quote unquote do it in the transformers head.”
Marcus: Pure transformer models lack mechanisms for truth and inherently hallucinate
“If you look at a completely pure case of a transformer model trained on a bunch of data, It doesn't have any mechanisms for truth. Now, except the sort of accidental contingency, and there are inherent reasons why these systems hallucinate, and maybe I can, in…”
Pesenti: Facebook AI transformer models matched or beat Mathematica at math
“My team also applied transformer models to mathematics, you know, doing, like, things like partial derivation equation or integration and showing that, hey, it could work as well better than Mathematica or some, you know, hundred page long algorithm.”