The Exchanges

Q So is this why you needed a parallel processing to be able to do that? Because you essentially have multiple layers operating at the same time, or no?

A Right. So the problem with this is that If you want the network to be able to recognize images properly or speech, you need them to be very, very large on the order of, um, so essentially the elementary operations that each of those, uh, elements are doing, uh, is, is, you know, multiply and, you know, multiplication by number and addition. And you may have, you know, in a typical conventional net, you may have something like between one and ten billion, uh, multiply, accumulate, operations, where each multiplication is a coefficient subject to learning. Ok. So you have a very large system. Computing the output takes, you know, five, ten billion operations. And you can't do this on the CPU. It's just too slow. So you have to go to GPUs. Current GPU cards are capable of, you know, four or five teraflops for a single GPU. You can parallelize on multiple GPUs, like something like four or eight in a single machine. And the problem, of course, is that you have to do this millions of times because you need to train the The network on millions of images before it's able to do any kind of proper recognition, and you have to cycle through those images perhaps a hundred times before it gets it right. So, uh, you know, the first kind of such systems, of course, convolutional nets have been, have been around for a long time, but the first large convolutional nets that are appropriate for o…

AI assessment note: “Right. So the problem with this is that If you want the network”

Q uh, of AI, at least for now? It's, uh, I guess we, we talk about Bell Labs. It's, so we go back to the Bell Labs model, but, um, I guess, how, how do you see that play out? The large companies are going to be the core, um, centers for, uh, research, and then if so, how do you contribute back, I guess, to the rest of the world?

A Right, ok, so I used to work at Bell Labs and NEC and, you know, a couple other, and I had friends at IBM and Microsoft Research and things like this, so I know a bit about industry or, uh, industry research. Um, uh, it's, Um, it's, it's a, it's a complicated, uh, issue. So it's not like everything is going to, you know, every interesting research is going to take place in industry and that, you know, academia is just going to be kind of watching us, uh, making progress. It's not like that at all. Uh, there are contributions that are very complementary from academia and industry. Uh, so of course industry has, you know, more computational resources, more data, uh, can get started quickly on, on, on projects because, you know, you're, you're next to your colleague who is one of the best specialists in the world for a particular technique. You just talk to your colleague and get started. You don't have to talk to anybody. Whereas in academia, you know, you, uh, have to get a grant. You hire students. You teach the students to, uh, uh, You know, get familiar with the techniques. Sometimes the students work out. Sometimes they don't. So you have to take another one. So every project takes longer to start up, ah, in academia. But, ah, but the type of motivation is different, and so the type of ideas that are generated is different. And I, I don't think at all that, ah, industry will…

AI assessment note: “We publish most of the stuff we do. We release most of our code”

Q um, do you, uh, want to build a, uh, very specific vertical solution, or do you want to build something horizontal? We had a, with a great company last time called X.AI that presented a very, um, you know, uh, focused AI-driven solution around scheduling. Right. Um, you know, there are companies, again, like Vicarious in the Valley that are building something much more sort of fundamental. Any, any thoughts?

A So I, I would think the, the verticals are probably better opportunities. So horizontal would be, you know, uh, here I have this, you know, awesome image recognition technology. Uh, but the thing is, you know, a lot of companies have that now. Uh, you know, this open source software you can download, you can train your own convolutional net on ImageNet, and you get a pretty good recognition performance. So, You know, what, what is the kind of value added that, that, that, that would be good for this? Uh, so I think, you know, verticals are probably, uh, probably a better idea. I'm actually a co-founder of a couple of companies. This is before I joined Facebook. I'm still sort of advising them. One that does biometrics using deep learning. Another one that does, uh, sort of music technology. So the second one is called Musami. Uh, it's been around for a while, and they use deep learning for various things, but they, and other techniques as well for kind of music-related consumer products and things like that. Um, or in apps, mobile apps. Uh, so, um, so things of that type where there is sort of a, a particular segment, uh, where, where, you know, there's a need. I think there's a huge amount of opportunities in things like medical imaging with deep learning. It's not very well explored right now. It's gonna take a while before the, so the research world sort of gets into it, and…

AI assessment note: “So I, I would think the, the verticals are probably better opportunities.”

Q So where's, um, where's deep learning going, I guess? So is, is, is one of the aspects going from supervised learning to unsupervised, um, which basically keeps getting closer and closer to, to the way a human brain works?

A Right, so the, um, the process I just described is supervised learning, which I'm, I'm sure many of you are familiar with, uh, when you do classification, it's supervised learning, right? So you have a data set composed of, A whole bunch of inputs with a whole bunch of corresponding desired outputs, and then you train the machine to map one to the other. Um, So, there is still progress to be made there, but we think we have pretty good handle with this. Um, I think, um, so on the side of applications, there's going to be this deep learning has sort of revolutionized speech recognition, uh, particularly a sub-module of speech recognition systems called acoustic modeling. So it hasn't really invaded the top high-level part of, uh, speech recognition system that consists in doing, uh, language modeling. So the thing that You know, contains the sort of knowledge about the, the language that's being, being spoken. Uh, but the acoustic modeling part, the part that maps the audio signal to kind of categories of sounds, if you want, that's done by, uh, the deep learning system. So that's completely taken over speech recognition over the last two years, uh, in a very, very short time. Uh, and, uh, uh, image recognition has been taken over a little later, um, uh, in the last year and a half or so. Um, um, and, uh, the next, there's a kind of a sense that in the, in the community that the…

AI assessment note: “the next set of techniques to kind of fall to deep learning... will be natural language”

Q Very, very cool. So I want to talk about AI in startups a little bit. So I guess the first question is, if you're a startup and you don't have all the data that, you know, Google and Baidu and Facebook have, do you have a fighting chance of building something?

A Well, uh, yeah, because, you know, there are data, there is data you can collect relatively quickly, uh, in, in sort of niche markets or even in, you know, sort of more, more generic things. Uh, the, the, I guess you're in luck if you, if you, if you've done a startup, you know, the, the window is going to close really quickly on, on deep learning, um, you know, sort of opportunity for startups, uh, in deep learning. Uh, the reason is that there are a bunch of companies That still don't have the, ah, that have data, and still don't have, kind of, the in-house competence for deep learning or expertise, and it's probably a good way for them to kind of get, ah, um, jump-started to just, you know, buy a startup that has some experience with it. Problem is that most of the startups, ah, you know, most of the students who had some experience with deep learning have either been hired by Google, Facebook, Microsoft, et cetera, or already sold their startup to, Twitter and whatever. Or haven't sold yet, but, you know, we'll see. Um, so the, the window is going to close, uh, very quickly because there are, uh, many people coming up on the market, uh, PhD students from, from various areas who have been exposed to deep learning, are starting to get a lot of experience, and so there's going to be, you know, a lot more supply than, than there was in the past. There was certainly a gold rush …

AI assessment note: “yeah, because, you know, there are data, there is data you can collect relatively quickly”

Q And, uh, so really just, just, um, leading all the way to, to Facebook, um, so, so you joined a year ago. What, what do you, what do you do at Facebook, and what does, uh, what, what does Facebook want to do with, uh, with deep learning?

A So, so Facebook's, uh, Facebook is 10, is 10 years old, um, and, um, or a little more, actually, coming to 11 years old. And, and Mark Zuckerberg, um, realized that Facebook was sort of relatively well established in its, uh, in its market. Pretty much dominant in the social network market. And started thinking about the next 10 years. What are the next 10 years going to be for, um, social interactions? And it's pretty obvious to a lot of people that a lot of our interactions, uh, you know, with our friends and a lot of interactions with the digital world is going to be mediated by AI systems. It's already the case that it's mediated by machine learning systems. So if you go to Facebook, uh, you know, every day we can, Facebook can show you, depending on how many friends you are, you are, you have, we can show you maybe 1500 to 2000 items. Posts, pictures, you know, news items, things like that. But of course nobody has time for this, uh, unless, you know, unless you are, I don't know, maybe a teenage girl spending all the time on Facebook, but, um, but, um, you know, most people don't have time for this. Although, Although about eight hundred million people connect to Facebook every day, different people connect to Facebook every day, and people on average spend 40 minutes per day on Facebook, so it's a lot of time. In fact, most people spend more time on Facebook on their pho…

AI assessment note: “interactions with the digital world is going to be mediated by AI systems”