The Ledger, every show
Every statement that passed quotation and attribution checks, across all 44 shows. Pick shows below, then mix any filter with any other.
shows 
every show 44 of 44
Morris: ChatGPT could likely have been built using RNNs instead of Transformers
“And I think like, we honestly probably could have gotten this with RNNs. I know like the scaling laws paper shows that RNNs have worse curves for scaling, but probably people would have been like, I bet you could have built chat GPT with a very sophisticated R…”
Jack Morris: Fundamental AI science shifted to companies due to academic compute limits
“That's when I think things really started to change in terms of the types of questions you wanted to ask can't always be answered with academic resources. So a lot of the like fundamental kind of like boundary pushing and AI science moved into companies.”
Morris: AI paradigm shifts are driven by novel datasets, not architectures
“I think, like, all of the things that I would consider paradigm shifts in the Kuhnian sense came from a new technique, but trained on new data, and I think the new data is super, super important”
Jack Morris: Most AI research was previously open, but is now closed
“Most stuff was open. Now most stuff is not open.”
Morris: Re-implementing new paradigm shifts quickly is the best AI grad strategy
“Honestly, if I were to give advice to a younger grad student, I think the way to do it would be literally just like sit and wait until the next kind of paradigm shift and then just immediately start working as fast as you can to like re-implement it. Like, I d…”
Morris: Two years of academic AI research on small models was inconsequential
“There was like kind of two years where everyone in academia was working on like smaller models and none of it really mattered.”
Morris: Machine learning lacks a fundamental unit of deep learning information
“I don't think we know what a bit is yet in terms of like deep learning models.”
Morris: New embedding inversion model exactly recovers 90% of source text
“Like we ended up building a system that can do this quite well, like taking an embedding and I think our highlight number is like at a certain length, like a long sentence length, we can get 90% of the text back exactly.”
Morris: Language models hit a hard memorization plateau regardless of dataset scaling
“Like, no matter how you scale the training size, you hit this like perfect, perfect ish plateau in auto memorization, which we call the model capacity.”
Morris: No Evidence We Can Build Pure Reasoning Models Without World Knowledge
“I don't think we have a lot of evidence that we can build a system like this that like is really, really good at reasoning, but really dumb about the world. Like, I don't know if we have the tools.”
Morris: Small models should be defined as runnable on a single GPU
“I think that we should establish the definition of small model as being a model that a grad student can inference at reasonable time on a single GPU. Which is probably like seven B maybe. I don't think 27 is small under any reasonable.”
Morris: 32-bit transformer models store only 3.6 to 3.9 bits per parameter
“Transformers that are trained in 32 bit precision, we approximate can store about 3.6 bits of information to maybe 3.9 bits somewhere in there per parameter.”
Morris: Open model creators do not use differential privacy or anonymization
“I would be extremely surprised if they do any type of like private training. Like there are these mechanisms for doing like differentially private language model training, or even just anonymization in the pre-training pipeline. I bet they don't do any of that…”
Morris: Weight deltas can reconstruct a competitor's proprietary fine-tuning dataset
“There's some tricks to it, but it's basically just like gradient based selection based on this weight difference. And it seems to be okay. Like it can get us pretty good training data. So I guess if you actually wanted to use this, it would be like your compet…”
Morris: The next AI paradigm shift will stem from an unused data source
“And so whatever the fifth thing is, whether it's Video or embodied AI or some kind of crazy innovation on reasoning models. Whatever comes next will probably be some type of new data source that we're not using yet.”
Morris: Top AI graduate programs do not teach multi-node distributed training
“Oh, to be clear, they don't teach you anything, like anything, like if you see a paper coming out from even, you know, Stanford, they're probably the best school in AI if you had to choose. And it's not like they're learning how to do like multi-node distribut…”
Morris: vLLM and SGLang are here to stay and will grow more complex
“I also think, ah, VLLM and SGLang seem, like, really good and important and here to stay. Like, they'll probably just get larger and more complex to accommodate future systems”
Morris: Information in AI should be measured under computational constraints
“There's this theoretical framework proposed in this paper, which is a theory of usable information under computational constraints from 20, 20. It really doesn't have that much press. They're not aren't as many citations as you would think, but I think it's a …”
Morris: Embedding inversion requires access to and repeated queries of the encoder
“Like none of the vector to text stuff works unless you have this assumption of like knowing the encoder and also being able to make a lot of queries to it.”
Morris: CycleGAN mapping aligns disparate model embeddings without paired data
“We took it and we applied it to model embeddings where instead of zebras and horses, we have like BERT embeddings and GPT embeddings, or like two completely different models with different architectures. So I think these are GTR, which is a T five based retrie…”
Morris: LLaMA architecture will likely store more information per parameter than GPT
“Maybe even if we tested this with LALAMA architecture, like, there's sort of like a GPT++ architecture, like, I would guess that can store better data just because the kind of numerical flow is a little bit better, the nonlinearities are maybe, like, A little …”
Morris: Deep understanding of GPU architecture makes engineers exceptionally hireable
“That said, if you do it, you're, you've gotta be one of the most hireable people in the world. Like if you like, Really deeply understand the architecture of the new GPUs coming out and how to control it. You're in a very small handful of people and like every…”