AI Research

“I think that we also talk about this like in the long, long term, that advantage that we have today will hopefully still be there, but it might not be as big on just the pure research. That's why the other components, that's why the product is so important. An…”

“A lot of AI research fails because the same model that writes the answer also grades the answer. That is like asking someone to write their own performance review and then being shocked when they describe themselves as a vision, a visionary.”

“There are a lot of researchers who just started out without formal training in machine learning or AI research. We've very much believed in training people up to do this. I think the real hard thing is the ability to creatively solve problems and think outside…”

“So, In that domain, I think it's happening now-ish. And I think we'll see the same thing too for AI research. That's a slower outer loop because now the experiment isn't just checking some unit tests passing, but it's checking what was the scaling property? Di…”

“But I think one of the interesting things that's come out of progress in artificial intelligence is we're seeing this big unification in neuroscience and AI. I think we're actually learning a lot from AI research, more than I think we thought we would learn fr…”

“In twenty-twenty-two was about the time when just Google completely stopped publishing its research. And so all the good papers are from before that as a result.”

“The number of ideas that actually work is not decreasing compared to the last, like, we're not in the era of diminishing returns yet.”

“I think over time that boundary has blurred quite a lot because we're working on these very large systems now. Research really looks like engineering and vice versa.”

“I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data unlimited regime where, where data would scale as much as you would like. And we're kind of shifting more to a data limited regime, which ac…”

“And this is how AI research tends to work, is there's some breakthrough that gets you this big step change function, and then there's actually a multi-year process of optimizing from there, where you get these kind of diminishing returns curves on breakthrough…”

“It's very different to be in a context of, you know, more deterministic software building. Than AI research. There are completely different motions of how you kind of run those teams in those areas.”

“We think that one of the most important signals of whether or not we are basically the speed of takeoff, the speed of progress is driven by how much AI is able to assist AI research.”

“I think almost everything kind of has to be done empirically. Like, you can kind of, like, come up with theories, but in practice, like, the first thing you're gonna do with your theory is test it, and most of the time you'll have gotten it wrong. So you shoul…”

“I think if you actually truly want to extend to research and, you know, finding, discovering ideas that, that meaningfully advanced technology on the, on, you know, the scale of like months and years, like I think the, these questions like stop being so differ…”

“Something I've said before and I'll say again is, is that data is the most under-invested in area of research relative to its impact, and I don't think it's even close.”

“Honestly, if I were to give advice to a younger grad student, I think the way to do it would be literally just like sit and wait until the next kind of paradigm shift and then just immediately start working as fast as you can to like re-implement it. Like, I d…”

“Sometimes fearless is this very interesting position where you're somewhat delusional and crazy, but somewhat just rationally bold, and it kind of is in between, because if you're too rational, it's not. You're not courageous enough. You're not identifying pro…”

“And there's so many like different components of research too. There's, it's not just you know, sitting off in the ivory tower thinking about things, but there's like hardware there's you know, various components of training and evaluation and stuff like this.…”

“Like they make, I think, still the best model suites for like doing research.”

“So that would be a tax on, on the NVIDIAs and AMDs and others, and it would be a tax on Meta and Amazon and all the folks that would have to pay that tax, which likely means that you get less chips purchased and less AI research, right? So if our number one go…”

“Just getting to that point where it's sort of plausibly begins to work is, is a huge, difficult problem. And it's completely separate from using scaling laws. Now, once you get it to work, that's when scaling laws come into play.”

“Fundamentally, I don't think, like, the, like, the stuff has actually, like, the underlying principles of research hasn't really changed that much, except for, like, compute.”

“During that twenty-twelve, twenty-eighteen era, the way people made progress what I mean by bottom-up basic research is you hire the most brilliant scientists, they come to work every day with, like, No near-term objective they're being held accountable to. An…”

“And so that's why, that's what I've been thinking about when I think in terms of AGI, can it speed up AI research? And yeah, I think that's like a plausible thing within the next five to 10 years.”

“The people making these models are AI researchers themselves, and I can imagine them being selectively trying to clearly they hear about their use case, which is helping them with their job, so I can imagine the model getting better at that than it gets better…”

“So I think in the world where you have really fast AI progress, and you are coming up to this point we're talking about where AIs can help improve themselves, then I think what you want to do is you might need a sort of government level actor to be like, all r…”

“There is this pressure to do things fast because AI is so fast, and the fastest thing to do is, oh, let's take what's working now and let's kind of, like, add on something to it, and that probably is, as you're saying, more general than just image to video but…”

“But for people in academia who have more time than money, they're getting completely frozen out of research. You can't do it. There's not enough money in all of academia to participate anymore in AI research”

“What we realized as we kept building those products was starting from like research that just like Became possible and identifying some coming, having some hypothesis of the ways in which it could really, we could build like really strong and valuable tools ar…”

“And when you do research, I think the failure is higher than, like, when you're building a product.”

“As these new tech companies have built these big research labs, they have pulled a lot of the good faculty out of the universities. And by the way, that process I think is accelerating in AI specifically because the capital needs to actually do modern AI are s…”

“When you're doing AI research, nobody really knows if it's going to take three months, or six months, or nine months, or 18 months, or if it's even possible.”

“Having a very clear Real world application for the research that you're doing makes it much more valuable than going off on a tangent and maybe being a bit too far ahead of the rest of the field.”

“We don't do AI research. We're not interested in any of that. We're looking at how can we kind of best take what is working in the state of the art and apply it rapidly into these defense problem sets.”

“It is to get lost, I would say, in the research terminology and the paper publication, on a specific number or metric that you want to improve, but if it doesn't really help your product, then, for us, it's, like, not really worth doing.”

“So I think most people in AI research think, no, we need an unknown number of further breakthroughs of principle, and we don't know what those are or when those would happen, and so we don't, we can't predict when we would have AGI, although there doesn't appe…”

“There's more AI research coming out of Europe than anywhere in the world. More than the US, more than China.”

“Increasingly at a lot of the conferences and looking at the papers that were being developed, it was more and more just big tech companies like ourselves that were showing up in the literature, and it was more and more a field becoming dominated by big players…”

“I'm very pro open publishing. Like, I think, like, it should be the default, and it's like, I'm still disputing situations where I'm like, you shouldn't publish on this stuff. Just because like, I think it is actually to the benefit of everybody to know what t…”

“And what people do is people kind of try to invent problems and the, such as solving some complicated games of character structure, and they try to Add kind of extra features to their models to combat those problems, but I think there's very little research ha…”

“And you know, it's not the kind of sexy research that people want to see, where you have like some hierarchy of big RNN, but it actually, this kind of research, I think at this point will advance field the most.”

“And I think what you're going to see is that there's a lot of work that's going to be done in games, but the goal is, of course, bring it out of the game, and actually use it to solve problems in the real world, and to actually, you know, be able to interact w…”