LLMs

“Google was basically the foundation of LLMs, right? They were too nervous and careful and risk averse to launching.”

“AI Town is kind of what this is going to end up Being. It's like, you need to give them the resources that they need to be pretty autonomous and to grow, and we're going to treat them more like peers, and they're going to talk to each other too, and it's more …”

“You can get to 90%, you just can't get to 99%, and this new technology really lets you get to 99%”

“You can imagine a world where LLMs get so good that you could feed in large portions of code written in COBOL, And they would spit out the same amount of code or the same functionality of said code in a different language, you know, and we're not there yet, bu…”

“LLMs can be a source of differentiation. Today, they might provide a first-mover advantage, but old-fashioned modes like network effects, high switching costs, scale, and brand are still what will likely win long term.”

“And everyone's supposed to do their own Turing test on every email they receive to see if it's real, which is not going to be sustainable. There probably isn't today, right? I mean, these LLMs are good enough to fake all of that voices, everything, right? So t…”

“We're public with the fact that we actually have LLMs in production at the agency. We have it in production in the open source team.”

“That's going to be an important principle for LLMs and AIs generally as they start agenting and causing change in the world. We have to care a lot about those boundaries.”

“For the past two-ish years or the past maybe three, four three years, let's say three years it's almost been more about execution than anything. It's a lot of just engineering, like how do you actually have large scale training work well. How do you make sure …”

“Like in LLMs, hallucination is a problem. So the fact that you're making shit up is a bad idea. But if you have to make shit up to convince someone, LLM is great and perfect.”

“There is this very interesting asymptotic kind of thing that's happening right now where, you know, two years ago, there was one, you know, LLM that was like way out ahead of everybody else's, which was opening eyes. And sitting here today, there's like six th…”

“So LLM's hallucinating, and for my consumer colleagues, that's a feature. It comes out with great, great creative, fun and enhancing things in the world of compliance that is definitively a bug.”

“LLMs are really good at making sense of unstructured data. So this is where we've seen them sort of most pervasive in the early innings of this movement.”

“The way to kind of just zoom out and think about this opportunity is like, where are there a bunch of humans that are basically dealing with whether it's voice or paper processes, where there's a bunch of unstructured data that they need to basically synthesiz…”

“These LLMs are going to replace search. But it turns out that's actually going to be really, really hard because there's a lot of things that search does that, that the models are bad at it really bad.”

“The piece about, ah, like secrets in code was some interesting research we did. Basically, we just went out and asked all the LLMs, write me an integration with GitHub, write me an integration with Stripe, and the vast majority of them hard coded the API key d…”

“For the most part, if you ask it to integrate with GitHub, it saw a plethora of different GitHub's keys and it's training data and it didn't regurgitate a specific one. It either regurgitate an example or like a put your thing in here, right?”

“The more you invest in docs and knowledge base, The more quality of the LLMs that you're going to have, like in terms of the output there.”

“If I look at a classic, say, compiler design or, you know, in, in, in, in programming languages, if I would have LLMs as a tool, I would probably think very differently about how I would build a compiler. And I don't think we've seen that work its way through …”

“We're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals, and that ends up masking all of these potentially undesired behaviors within the system itself.”

“And then out of nowhere comes these LLMs, and they are unit economic positive, they solve all of these language problems, like, basically immediately.”

“And that is going to be a lot of investment that you will see across the board, which is how do we build the right systems around AI and LLMs that solves the problem.”

“The underlying infrastructure we have is focused on being able to rip and replace any model, the best one that's out there, and sometimes use multiple models from different vendors after fine tuning them with our hundreds of millions of utterances that are lab…”

“Our customers don't care what models we use. They want to make sure they can deliver a better, faster, and more proactive experience to their patient populations.”

“I think LLMs are now kind of in this, like, scaling paradigm, right, where, like, people are putting more compute in, they're doing more, like, RLHF, they're doing more reasoning. But, like, you know, video is actually still really early.”

“When you're able to bring the cost of this technology down low enough, I think it actually is fundamentally going to change how people interact with these LLMs that are driving a lot of these new experiences.”

“All, like, all LLMs, but I'll use ChatGPT as the example, because the most people use it have this really unfortunate and annoying problem of hallucinating around product recommendations that basically everyone experienced if you tried to use it for that.”

“And we feel like teaching these LLMs to be foundation models, but for quantum mechanics will be the next frontier for LLMs.”

“I think far enough that we can make something that will figure out the next breakthrough with the current technology. Like I, it's a very self-referential answer, but if LLMs can get, if LLM based stuff can get far enough that it can do like better research th…”

“There has to be an architectural lead that is able to create these manifolds and just throwing new data will not do it. It'll just smoothen out the already existing manifolds.”

“They reduce a very, very complex multidimensional space into Basically a geometric manifold that's a reduced state space. So it's reduced degrees of freedom, but you can actually predict where in the manifold the reasoning can move to.”

“But the thing that people on track is it's going to be LMS and diffusion models. And I think other things with a fabric across them.”

“The same thing I think is going to happen in LLMs. At first, you have one enterprise use one big LLMs. Now they're going to use five, 1000, small LLM, medium, large, whatever. Once you get to, you don't do tokens re-limiting, token authentication in each LLM u…”

“I would say LLMs today, you know, they're marketed as a million token length. Which is like a million words almost. In reality, it's about 200,000, and then they start to struggle.”

“I don't think so. I mean, I think there are certain things like memory and learning, like continuous learning that are not very easy with the current architectures. I think even those you can sort of fake and maybe are, we're going to be able to get them to wo…”

“It just does, it does feel like the LLMs in a way are distracting from that because all the talent is going there. And therefore there's less talent that are trying to do basic research on, on intelligence.”

“Another related problem is that since we're Dependent on, ah, expert data in order to train the alums and the alums start to substitute those workers. But, you know, at some point there's no more experts because they're all out of jobs and they're equivalent t…”

“I do think at some point you get to LMs can, they can do Everything, every single thing a human can do for cheaper. Like, I don't see a reason why we don't eventually get there. That may take five, 1015 years.”

“CS majors graduating from college, there's just not as many jobs as there used to be, and LLMs are a little more substitutable for what they previously would have done, and I'm sure that's contributing to it.”

“In the true pre-training scaling era, you know, GPT-II, three, 3.5, maybe up to four it felt like you can just put more internet data in there and just, it just got better. Whereas now it feels like there's a lot of labeling work happening, there's a lot of co…”

“I don't think LLMs as they can understand are on, on the way to AGI and my definition for AGI is I think the old school RL definition, which is a machine that can go into any environment and learn efficiently in the same way that a human could go into you can …”

“And as a result, a lot of the techniques we use are like, basically we're just deeply under regularized. Like the models are super overfit. The clever trick is they're overfit on the domain of all of human knowledge, which turns out to be a pretty awesome way …”

“Because the models are so hard to abstract away. Like, they're just unruly, right? If you try to, like, have traditional software drive them, they just don't kind of manage very well. So part of me thinks that it's almost like this, like, anti-disintermediatio…”

“If you look at all possible combinations of 8000 tokens and 50,000, ah, vocabulary, the number of rows In this matrix is more than the number of electrons across all galaxies, right? So, so there's no way that these LLMs can represent it exactly.”

“You can rule out they're conscious. I mean, come on. And I said, you know, Anthropic makes great products. Cloud code is fantastic. Co-work is fantastic, but they are grains of silicon doing matrix multiplication. They don't have consciousness. They don't have…”

“So in kind of an abstract way, what all these LLMs are doing is coming, coming up with a compressed representation of this matrix. And when you give a prompt, They try to approximate what the true distribution should have been and try to generate it.”

“No. Actually, the opposite.”

“LLMs are going to be very good at anything where you can describe how people do it. And where what you want is the way anybody would do that, and not so good at where you can't really explain why you did it like that, and where you're doing it differently to t…”