Schönlieb: Exactly minimizing training loss often hurts neural network generalization
Carola Schönlieb · Mathematical Approaches to Image Processing with Carola Schönlieb · Y Combinator · May 9, 2018 · at 23:20
Applied mathematician Carola Schönlieb explains why stochastic optimization and approximate loss minimization are preferred in machine learning over exact training loss minimization.
0:00 / 0:39exact quote · 39.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You do not necessarily need to solve your optimization problem, your training exactly. And maybe sometimes, or most of the time you actually don't want it, want to save it exactly because you only have a finite amount of training examples. And so when you think about what these neural networks are doing, they're trying to minimize a loss over the training examples that you have. But this loss is only an approximation of many, many, many, many more images that you want your neural network to work for. And so very often you do not want to solve that exactly. You don't want to minimize your loss exactly for this training set.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Carola Schönlieb
Medical AI models trained on one scanner brand fail on others
“Also the type of scanner you're using, are you using a G or a Siemens or Toshiba or whatever they have different settings, And they have different ways of going from the measurements to an image. And so, you know, if you train an algorithm, for instance, a neu…”
Schönlieb: Deep neural networks outperform handcrafted methods in image denoising
“Image denoising nowadays, I think the best image denoising approaches are actually coming from deep neural networks. So, you know, these handcrafted methods get more and more beaten in terms of performance. By some of these neural network approaches.”
Schönlieb: ML super-resolution creates probable images but cannot verify accuracy
“Maybe, you know, if you have all these machine learning methods which have learned to look at just pixels and then know what is a very probable match in terms of high resolution, maybe at some point you can do it, but then you don't know how, if you're right o…”
Schönlieb: Photoshop's Content-Aware Fill is based on mathematical research
“And I mean, also the content-aware fill is actually very much based on some of the things that have been initiated by people like Andrea Batozzi. So, I mean, the technique is Different in what Photoshop is using, but it's still based on research in mathematics…”
Schönlieb: Handcrafted image models provide mathematical proofs and error estimates
“We can prove properties about the denoising abilities of these methods of how stable they are, for instance, to perturbations in the images. We know how that works, so we can prove things about that. We have error estimates and things like this.”
Schönlieb: CT reconstruction is constrained by limits on patient radiation exposure
“You want a very high resolution image because you want to look at all the details in the body. But you don't want to measure so many line integrals because you don't want to radiate the patient so much. You don't want to send tons of x-rays through, through th…”