# Intuition or interpretation of the first term of the Mumford Shah functional in image processing

I was hoping someone could explain the intuition behind the first term of the Mumford-Shah functional, used in image segmentation problems. I have been watching an interesting video series on variational methods in image processing, but the presenter kind of glosses over the explanation of the terms in the Mumford-Shah functional.

The form of the M-S functional (taken from Wikipedia) is:

$$E [ J , B ] = C \int _ { D } ( I ( \vec { x } ) - J ( \vec { x } ) ) ^ { 2 } d \vec { x } + A \int _ { D / B } \vec { \nabla } J ( \vec { x } ) \cdot \vec { \nabla } J ( \vec { x } ) d \vec { x } + B \int _ { B } d s$$

This first term $$(I(x) - J(x))^2$$ is unclear to me. According to Wikipedia $$I(X)$$ is the image and $$J(x)$$ is an image model. But I am not clear on what is meant by the "image model" versus the image itself? Is there some prediction of the pixel intensities that creates a model $$J(x)$$ and then the first term compared the predicted intensities versus the actual intensities in $$I(x)$$. I was not clear on how the model $$J(x)$$ was computed. Is it a regression, or a local average over the domain, or such.

If anyone can explain this first term in the model, or point me to a good explanation, that would be really helpful.

Also note, I did read this related Stackexchange post, but it did not really provide enough intuition.

The image function in Mumford-Shah functional in image segmentation

## 1 Answer

If you look at Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems by Mumford and Shah, you'll see that the idea is to segment an image into a decomposition of disjoint regions $$\{R_j\}$$ such that $$I(x)$$ varies slowly within each region $$R_i$$, but quickly across the boundaries between two regions. The idea is to use some piecewise smooth function $$J$$ to approximate the image $$I$$. The point of the first term is to be able to use this approximation to match the image on a single region. Assuming $$J$$ on some region is very smooth, then $$I$$ on that region must also be smooth if term 1 is small. The idea is that an image segment should be smooth; e.g., it should include the surface of an apple but not allow the decidedly non-smooth discontinuous jump to the white background wall behind it. In other words, term 1 enforces smoothness within each region (i.e., intra-region self-similarity).

The exact form of this approximation is left vague, as it is task-dependent. Being piecewise constant, or only allowing linear variability on each region, as examples, are all reasonable, valid options, but it can also be more powerful functions. However, obviously a powerful $$J$$ can arbitrarily closely approximate $$I$$, meaning term 1 will shrink to zero. What exactly, then, enforces smoothness in a single region in this case? Well, term 2 does. If $$J$$ becomes too complex (i.e., too powerful, capable of representing a highly discontinuous region to shrink term 1), then $$||\nabla J||_2^2$$ will increase. Term 2 also allows us to sidestep the issue of the exact form of $$J$$, because it penalizes excessive "complexity" regardless of its exact parameterization.

But there is a hole in having just term 1 and 2. Sure, a single region cannot be too non-smooth, but what stops us from having many tiny regions? This would let us shrink term 1 without incurring a penalty from term 2 by cutting out every slowly varying region no matter how tiny it is. This is accomplished by term 3, which penalizes the total length of all inter-region boundaries, preventing the presence of many small boundaries.