May 9, 2018 · 41m · y-combinator
Mathematical Approaches to Image Processing with Carola Schönlieb · Y Combinator
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Applied mathematician Carola Schönlieb discusses the evolution of mathematical image processing, explaining how integrating classical physics-informed equations with modern deep learning provides robust, verifiable solutions across medicine, environmental monitoring, and art restoration.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The partners as informed peer 2.5 Guest teaching 5.9 Guest disagreement 0.3 The partners pushing back 0.3
speaking balance: gold is the partners, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:21 Calling out Hollywood crime show image enhancements
Schönlieb dismisses the fictional trope of magically zooming into pixelated security footage, pointing out the inherent mathematical impossibility and the danger of neural network hallucinations.
Hardest push from the partners ▶ 15:21 Challenging scanner variance assumptionsCannon challenges the premise that different CT hardware creates distinct data distributions, assuming scanners from different brands were essentially identical inside.
Biggest teaching moment ▶ 26:04 Demystifying generalization and loss function minimizationSchönlieb thoroughly breaks down why perfect empirical loss minimization on a finite training set leads to overfitting rather than true generalization across infinite data distributions.
The partners hold their own ▶ 11:45 Connecting mathematical edge preservation to Photoshop compositingCannon demonstrates practical domain familiarity by explaining how blurring cutouts was essential at The Onion to match camera depth of field and avoid obvious compositing artifacts.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The partners as informed peer | Guest teaching | Guest disagreement | The partners pushing back | Why |
|---|---|---|---|---|---|---|
| Early Background in Mathematics and Partial Differential Equations | 1 | 5 | 0 | 0 | Cannon asks introductory questions about Schönlieb's research background. Schönlieb explains partial differential equations, the Cahn-Hilliard equation, and stability analysis in metallic alloys in an accessible academic manner. | |
| Transition to Image Restoration and Inpainting | 3 | 4 | 0 | 0 | Cannon makes a relevant connection to Photoshop's content-aware fill tool. Schönlieb confirms the connection and explains how mathematical PDEs underpin inpainting and image restoration techniques. | |
| Inverse Problems and Computed Tomography Reconstruction | 2 | 6 | 0 | 0 | Schönlieb details inverse imaging problems and Radon transforms in CT scanning. Cannon asks how data missingness and denoising are addressed in reconstructing 3D images from 2D line integrals. | |
| Edge Preservation and Mathematical Denoising Principles | 4 | 6 | 0 | 0 | Cannon references audio Fourier transforms and visual compositing at The Onion. Schönlieb educates on why total variation and edge preservation are crucial to prevent blurring sharp color boundaries. | |
| Handcrafted Models vs Deep Learning and Scanner Variances | 2 | 7 | 1 | 1 | Cannon is surprised that CT scanners from different manufacturers output subtle variances that break neural models. Schönlieb explains adversarial fragility, structural parameter reduction, and model interpretability. | |
| Hybrid Mathematical-ML Approaches and Optimization Theory | 3 | 7 | 0 | 1 | Cannon presses for clarification on why exactly minimizing loss during training can be detrimental. Schönlieb breaks down stochastic optimization, generalization bounds, and bi-level optimization. | |
| Real-World Applications in Medicine, Fluid Dynamics, and Forestry | 2 | 6 | 0 | 0 | Schönlieb details applied collaborations in hospital MRI scanning, fluid dynamics, and forestry using hyperspectral and LIDAR data. Cannon follows along with examples from archeology documentaries. | |
| Surveillance, Image Enhancement Tropes, and Machine Learning Hallucinations | 3 | 6 | 1 | 0 | Schönlieb debunks Hollywood 'zoom-and-enhance' tropes by explaining the risk of ML hallucination, then describes virtual restoration of illuminated manuscripts at Cambridge. Cannon provides an art forgery anecdote. |
Statements from this episode (9)
Schönlieb: Photoshop's Content-Aware Fill is based on mathematical research
“And I mean, also the content-aware fill is actually very much based on some of the things that have been initiated by people like Andrea Batozzi. So, I mean, the technique is Different in what Photoshop is using, but it's still based on research in mathematics…”
Schönlieb: CT reconstruction is constrained by limits on patient radiation exposure
“You want a very high resolution image because you want to look at all the details in the body. But you don't want to measure so many line integrals because you don't want to radiate the patient so much. You don't want to send tons of x-rays through, through th…”
Cannon: Inconsistent Edge Sharpness Makes Photoshopped Images Easy to Spot
“One of the most obvious things to spot a Photoshop is a sharp edge and a soft edge in the same photo.”
Schönlieb: Deep neural networks outperform handcrafted methods in image denoising
“Image denoising nowadays, I think the best image denoising approaches are actually coming from deep neural networks. So, you know, these handcrafted methods get more and more beaten in terms of performance. By some of these neural network approaches.”
Medical AI models trained on one scanner brand fail on others
“Also the type of scanner you're using, are you using a G or a Siemens or Toshiba or whatever they have different settings, And they have different ways of going from the measurements to an image. And so, you know, if you train an algorithm, for instance, a neu…”
Schönlieb: Handcrafted image models provide mathematical proofs and error estimates
“We can prove properties about the denoising abilities of these methods of how stable they are, for instance, to perturbations in the images. We know how that works, so we can prove things about that. We have error estimates and things like this.”
Schönlieb: Exactly minimizing training loss often hurts neural network generalization
“You do not necessarily need to solve your optimization problem, your training exactly. And maybe sometimes, or most of the time you actually don't want it, want to save it exactly because you only have a finite amount of training examples. And so when you thin…”
Schönlieb: ML super-resolution creates probable images but cannot verify accuracy
“Maybe, you know, if you have all these machine learning methods which have learned to look at just pixels and then know what is a very probable match in terms of high resolution, maybe at some point you can do it, but then you don't know how, if you're right o…”