Assertion Not publicly verifiable
Howard: Alec Radford Built OpenAI's GPT After Reading ULMFiT
“I organized a chat for both of us with Kate Metz in the New York Times, and Kate Metz answered, sorry, and Alec answered this question for Kate, and Kate just like, so how did, you know, GPT come about? And he said, well, I was pretty sure that pre-training on…”
Opinion
Howard: Meta 'blew it' on Code Llama due to catastrophic forgetting
“So Code Llama was a, I think it was like a five hundred billion token fine-tuning of Llama II using code. And also prose about code that Meta did. And honestly, they kind of blew it. Because Code Llama is good at coding, but it's bad at everything else.”
Opinion
Howard: TensorFlow 2 was a failure that Google avoided internally
“Then in the end, you know, Google didn't follow through, which is fair enough, like, asking everybody to, you know, learn a new programming language is going to be tough, but, like, it was very obvious, very, very obvious at that time that TensorFlow II was go…”
Assertion Not checkable as stated
Howard: JAX was a grassroots Google reaction against TensorFlow 2
“But I mean, in the meantime, I will say, you know, Google now does have a backup plan. You know, they have JAX, which was never a strategy. It was just a bunch of people who also recognized TensorFlow two as shit, and they just decided to build something else.”
Opinion
Howard: RAG is an inefficient hack compared to fine-tuning
“RAG is like such a inefficient hack, really, isn't it? It's like, You know, segment up my data in some somewhat arbitrary way, embed it, ask questions about that, you know, hope that my embedding, you know, model embeds questions in the same embedding space as…”
Assertion Not checkable as stated
Howard: Answer.AI runs fully in-house stack without AWS or Google Cloud
“This group of, which has averaged about 10 to 12 people, currently nine, I think, have built a pretty Transformational and complex piece of software, which we can do a quick demo of later if you're interested. Using a complete web application development platf…”
Insight
Howard: Correcting LLM errors in chat history degrades subsequent model answers
“The autoregressive nature of language models means that if they make a mistake, and you correct it, and then say, no, that was a mistake, please do it this way instead. The more often you do that, the worse the dialogue answers get. Because it's in the trainin…”
Insight
Howard: Longer dialogues improve AI outputs when humans edit intermediate results
“The nice thing is that with the dialogue engineering we discussed, the longer your dialogue is, the better the AI gets, which is the opposite to what we're used to, right? Because you can edit the outputs that aren't great.”
Insight
Howard: Training AI models from random weights is almost never justified
“If you're training for random weights, you better have a really good reason, you know, because it seems so unlikely to me that nobody has ever trained on data that has any similarity whatsoever to the general class of data you're working with, and that's the o…”
Insight
Howard: Pre-training data mixes should be continuous per-batch functions, not discrete phases
“So the point at which they're doing proper continued pre-training is the point at which that becomes a continuum rather than a phase. So the only difference with what I was describing last time is to say, like, oh, they should, you know, There's a function or …”
Insight
Howard: Non-profit boards cannot control commercial entities with equity-compensated staff
“This didn't make sense to have like a so-called non-profit where then there are people working at a commercial company that's owned by or controlled nominally by the non-profit where the people in the company are being given the equivalent of stock options. Li…”
Insight
Howard: Corporations are sociopathic by design due to fiduciary duty
“Companies are sociopathic, like, by design. And so the alignment problem, as it relates to companies, has not been solved. Like, companies become huge, they devour their founders, they devour their communities, and they do things where even the CEOs, you know,…”
Assertion Not checkable as stated
Howard: 80% of top unique creators have unconventional or non-mainstream backgrounds
“Like, 80% of the time, I find out the person has a really unusual background. So, like, often they'll have, like, either they, like, came from poverty and, like, didn't get an opportunity to go to good school, or they, like, you know, had dyslexia and, you kno…”
Disclosure
Howard: Answer.ai operates with no managers and zero corporate hierarchy
“We don't have any managers. We don't have any hierarchy from that point of view. So, for example, I'm not a manager, which means I don't get to tell people what to do or how to do it or when to do it.”
Opinion
Howard: Tech builds too many vanity foundation models over fine-tuning
“People are building too many vanity foundation models rather than taking better advantage of fine-tuning”
Assertion Not checkable as stated
Howard: Decoder models must be far larger to match DeBERTa
“Now, the interesting thing is, you see, unlike Kaggle competitions, that decoder models still Are at least competitive with things like DiBerta VIII. But they have to be way bigger to be competitive with things like DiBerta VIII. And the only reason they are c…”
Prediction Not checkable as stated
Howard: Reka's model is probably superior to GPT and Claude for certain tasks
“There's a whole model that's been trained in a different way. So there's probably a whole lot of tasks it's probably better at than you know, GPT and Gemini and Claude.”
Insight
Howard: Developers should distribute merged adapters rather than merged models
“To explain, it's not that you shouldn't merge models, it's that you shouldn't be distributing a merged model. You should distribute it a merged adapter. 99% of the time. And actually often, one of the best things happening in the model merging world is actuall…”
Disclosure
Howard: Answer.ai aims to build thousands of products with 12 people
“We want to create thousands of Commercially successful products at Answer.ai. And we want to do that with like, 12 people.”
Opinion
Howard: Building web apps is much worse now than 15 years ago
“Much to my, you know, horror, the story around creating web applications is much worse now than it was 10 or 15 years ago, in terms of, like, if I say to a data scientist, here's how to create and deploy a web application, You know, either you have to learn Ja…”
Opinion
Howard: Cursor and VS Code shoehorn AI into legacy software paradigms
“It's like a convenience over the top of this incredibly complicated system that full-time, sophisticated software engineers have designed over the past few decades in a totally different environment as a way to build software, you know. And so we're trying to,…”
Insight
Howard: Accurate next-token prediction forces models to learn world models and causality
“I thought, okay, so if I do this at a much bigger scale, using all of Wikipedia, what would it need to be able to do to finish a sentence in Wikipedia effectively, to do it quite accurately, quite often? I thought, geez, it would actually have to know a lot ab…”
Assertion Supported
Howard: Experiments show LLMs can memorize full datasets in one epoch
“And so we ran a bunch of experiments, and all of them supported the hypothesis that it was memorizing the data set in a single thing at once.”
Opinion
Jeremy Howard: The three-step ULMFiT fine-tuning approach is wrong and obsolete
“Even though I originally created the three-step approach that everybody now does, my view is it's actually wrong, and we shouldn't use it.”
Jeremy Howard