I was hoping someone could explain the intuition behind the first term of the Mumford-Shah functional, used in image segmentation problems. I have been watching an interesting video series on variational methods in image processing, but the presenter kind of glosses over the explanation of the terms in the Mumford-Shah functional.

The form of the M-S functional (taken from Wikipedia) is:

$$ E [ J , B ] = C \int _ { D } ( I ( \vec { x } ) - J ( \vec { x } ) ) ^ { 2 } d \vec { x } + A \int _ { D / B } \vec { \nabla } J ( \vec { x } ) \cdot \vec { \nabla } J ( \vec { x } ) d \vec { x } + B \int _ { B } d s $$

This first term $(I(x) - J(x))^2$ is unclear to me. According to Wikipedia $I(X)$ is the image and $J(x)$ is an image model. But I am not clear on what is meant by the "image model" versus the image itself? Is there some prediction of the pixel intensities that creates a model $J(x)$ and then the first term compared the predicted intensities versus the actual intensities in $I(x)$. I was not clear on how the model $J(x)$ was computed. Is it a regression, or a local average over the domain, or such.

If anyone can explain this first term in the model, or point me to a good explanation, that would be really helpful.

Also note, I did read this related Stackexchange post, but it did not really provide enough intuition.

The image function in Mumford-Shah functional in image segmentation


If you look at Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems by Mumford and Shah, you'll see that the idea is to segment an image into a decomposition of disjoint regions $\{R_j\}$ such that $I(x)$ varies slowly within each region $R_i$, but quickly across the boundaries between two regions. The idea is to use some piecewise smooth function $J$ to approximate the image $I$. The point of the first term is to be able to use this approximation to match the image on a single region. Assuming $J$ on some region is very smooth, then $I$ on that region must also be smooth if term 1 is small. The idea is that an image segment should be smooth; e.g., it should include the surface of an apple but not allow the decidedly non-smooth discontinuous jump to the white background wall behind it. In other words, term 1 enforces smoothness within each region (i.e., intra-region self-similarity).

The exact form of this approximation is left vague, as it is task-dependent. Being piecewise constant, or only allowing linear variability on each region, as examples, are all reasonable, valid options, but it can also be more powerful functions. However, obviously a powerful $J$ can arbitrarily closely approximate $I$, meaning term 1 will shrink to zero. What exactly, then, enforces smoothness in a single region in this case? Well, term 2 does. If $J$ becomes too complex (i.e., too powerful, capable of representing a highly discontinuous region to shrink term 1), then $||\nabla J||_2^2$ will increase. Term 2 also allows us to sidestep the issue of the exact form of $J$, because it penalizes excessive "complexity" regardless of its exact parameterization.

But there is a hole in having just term 1 and 2. Sure, a single region cannot be too non-smooth, but what stops us from having many tiny regions? This would let us shrink term 1 without incurring a penalty from term 2 by cutting out every slowly varying region no matter how tiny it is. This is accomplished by term 3, which penalizes the total length of all inter-region boundaries, preventing the presence of many small boundaries.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.