Convolution intuition: clarifying Terence Tao's “blurring”/“fuzz” interpretation

On this math.MO post, "What is convolution intuitively?", Terence Tao's answer (in the case where one function is a bump function) involves "blurring" and "fuzz."

Could someone clarify his interpretation more explicitly? The intuition still escapes me. Thanks!

• Have you already seen en.wikipedia.org/wiki/Convolution#Derivations ? There you can find some animations which might be helpful for understanding. – Ruslan Jul 23 '13 at 9:54
• @Ruslan Yes, I've stared at those animated gifs before, and I sort of understand the "sliding area" interpretation. How does it relate to Tao's fuzz interpretation? – angryavian Jul 23 '13 at 10:23
• If you want to blur a digital image (represented as an $m \times n$ matrix $A$), you can do that by convolving $A$ with a Gaussian blurring kernel $B$. You could try it in Matlab and look at the results. And you could compute the convolution by hand. You'll see how the blurring happens -- each pixel in the convolution is just a weighted sum of nearby pixels in the original image. – littleO Jul 23 '13 at 10:40
• What exactly confuses you about this answer? I doubt I can write a better explanation than Terrance Tao of his own mathematical intuition, so if you want something clarified, you have to be specific. – Alexander Gruber Jul 23 '13 at 13:49
• @AlexanderGruber I was looking for a something more concrete like Ruslan's answer below. What still eludes me though is his statement about convolution being the "fuzzy version of addition." I see that a bump function will "blur" another function everywhere, but what is the addition that Tao mentions? – angryavian Jul 23 '13 at 19:41

Consider a function $f_1$: $$f_1(x)=\delta(x)$$ where $\delta(x)$ is Dirac delta, and a Gaussian function $g(x)$: $$g(x)=\frac1{\sigma\sqrt{2\pi}}\exp\left(-\frac{x^2}{2\sigma^2}\right).$$ One of main properties of Dirac delta is this: $$\int_{-\infty}^\infty q(x)\delta(x-x_0)dx=q(x_0).\tag1$$ Thus, convolution of $f_1$ and $g$ will equal $g$. Let's see how it looks ($f_1$ is red, $f_1*g$ is blue): You can see the 'blurring' effect of gaussian convolved with a Dirac delta. Now consider more complex function: $$f_2(x)=\sum_{i=-N}^N a(x_i) f_1(x-x_i)\Delta x_i.\tag2$$ $\Delta x_i$ here is step in $x$, i.e. $\Delta x_i=x_i-x_{i-1}$. We'll need it later. We'll take $\Delta x_i=\Delta x_j\; \forall x,j\in\mathbb{Z}$, then it can be viewed as just a coefficient before sum.

Here's a plot of $f_2$ and $f_2*g$ ($f_2$ is red, $f_2*g$ is blue): Here you can also see that the result of convolving a gaussian with a sum of Dirac deltas gives you blurred version of original function. It's easy to generalize $f_2$ to a sum of infinite number of deltas with $N\to\infty$: $$f_{2a}(x)=\sum_{i=-\infty}^\infty a(x_i)f_1(x-x_i)\Delta x_i.$$

$f_{2a}(x)$ resembles a Riemann sum. Let's use this and take the limit $\Delta x_i\to0$: $$f_3(x)=\lim_{\Delta x_i\to 0}f_{2a}(x)=\int_{-\infty}^\infty a(t)f_1(x-t)dt\equiv\int_{-\infty}^\infty a(t)\delta(x-t)dt.\tag3$$ From $(1)$, $$f_3(x)=a(x).$$ In $(3)$ after taking limit $x_i$ becomes $t$, and $\Delta x_i$ becomes $dt$. $\Delta x_i$ is needed to gradually attenuate Dirac delta amplitude as $\Delta x_i\to 0$, so that in the limit it becomes finite, resulting in a finite function.

Here's a plot of $f_3$ and $f_3*g$ ($f_3$ is red, $f_3*g$ is blue): This is finally the convolution of a usual function with a gaussian, which still does resemble the blurring, but is somewhat less obvious.

Also note that since $p*q=q*p$, you could as well start from a Dirac delta, represent $g(x)$ as a sum of scaled&translated $\delta(x)$'s and get the same result by summing $a(x)$'s scaled by $g(x_i)$ instead of $g(x)$'s scaled by $a(x_i)$.

• The pictures help a lot, but the equations $f_2$ and $f_3$ confuse me more. I see that $f_2$ is just the sum of Dirac delta functions translated by $x_i$ and scaled by constants $a_i$, but what is $\Delta x_i$? In $f_3$, what is the function $a(x)$? And why did you choose to make $f_3(x)$ look like another convolution, even before convolving with $g(x)$? – angryavian Jul 23 '13 at 19:36
• Thanks Ruslan, that clarifies a lot! One more question: what do you mean by "normal"/"usual" function? Are you saying that any function can be written in the form $f_3$? – angryavian Jul 24 '13 at 7:20
• I've edited my answer to make it a bit more consistent. So I think I'll remove my previous comments (if you agree). As for your question about "normal" function, I'd say yes, any function can be represented in this form because of $(1)$. But I might be beaten by someone who suggests a pathological case for which $(1)$ is untrue :) – Ruslan Jul 24 '13 at 8:09
• looks good to me! – angryavian Jul 24 '13 at 8:44

I'll just answer this comment, since the rest seems to have been addressed satisfactorily.

First we need to understand the statement "functions are fuzzy versions of points". What this is saying is that the support of a function is not a single point, but many. Consider $f$ being a distribution of compact support, which we write $f\in \mathcal{E}'$ (if you are not familiar with distribution theory, just think locally integrable functions with compact support for now). Since the coordinate projections $(x_1,\ldots,x_n)\mapsto x_i$ are $C^\infty$ functions, we have that the pairing $$\langle f,x_i\rangle \tag{CoM}$$ is well defined. (In terms of locally integrable functions, think of equation (CoM) as $\int_{\mathbb{R}^n} x_i f(x) \mathrm{d}x$. Since $f$ has compact support the integral converges.)

This allows as to define the center-of-mass for each $f\in \mathcal{E}'$. That is, let $\mu: \mathcal{E}' \to \mathbb{R}^n$ be the mapping $$\mu(f) = \frac{1}{\langle f,1\rangle} \left( \langle f,x_1\rangle, \langle f,x_2\rangle,\ldots,\langle f,x_n\rangle\right)$$

Quite obviously $\mu$ is not injective: multiple functions/distributions can have the same center of mass. It is in this sense that functions are "fuzzy versions" of points: think of $f\in \mathcal{E}'$ with mass $\langle f,1\rangle = 1$ as something that is like the point $\mu(f)$, but spread out a bit in space.

Now let $f,g\in \mathcal{E}'$. Their convolution is well-defined. (See the Wikipedia link above.) We have the characterisation that for any smooth function $\phi$ the convolution satisfies $$\langle f*g,\phi\rangle = \langle f,\psi\rangle$$ where $$\psi(y) = \langle g,\tau_{-y} \phi\rangle$$ In terms of integrable functions this just says that $$f*g(x) = \int f(x-y)g(y) \mathrm{d}y$$ the usual definition.

Now, a direct computation shows that $$\mu(f*g) = \mu(f) + \mu(g)$$ This you can easily check using the integrable functions definition also: $$\int x_i f*g(x) \mathrm{d}x = \iint x_i f(x-y) g(y) \mathrm{d}y\mathrm{d}x = \iint (z_i + y_i) f(z) g(y) \mathrm{d}z \mathrm{d}y = \int g(y) \mathrm{d}y \int f(z)z_i \mathrm{d}z + \int f(z) \mathrm{d}z \int y_i g(y) \mathrm{d}y$$

That is to say: when you convolve two functions/distributions, you add their center of mass. So if you treat $f\in \mathcal{E}'$ as fuzzy versions of points, the convolution is their natural law of "vector addition".

Lastly know that the Dirac delta function $\delta_x\in \mathcal{E}'$. They correspond to the actual sharp (non-fuzzy) points!

• Thanks for the in-depth explanation! The distribution theory bits escaped me, so thank you for including the integrable functions interpretation as well. A few more questions: 1) I think I get the "adding centers of mass" idea. Is there a way to describe the shape of the convolution in terms of the fuzz of $f$ and $g$? 2) I'm guessing that the mass is preserved, in that $\langle f*g,1\rangle=1$. Is this true? – angryavian Jul 25 '13 at 13:35
• (1) Yes, the support set of $f*g$ will be a subset of the Minkowski sum of their supports. When $f$ and $g$ are both non-negative, the two will in fact be equal. (When the functions are indefinitely-signed, there can be occasional cancellations.) (2) I am not sure what you mean by preservation of mass. Are you assuming that $\langle f,1\rangle = \langle g,1\rangle = 1$? In which case yes, whenever interchanging order of integration is allowed. – Willie Wong Jul 29 '13 at 8:35
• In general one expects (when one can interchange order of integration; see Fubini's theorem for example) that $$\int f*g(x) \mathrm{d}x = \iint f(x-y) g(y) \mathrm{d}y \mathrm{d}x = \int f(z) \mathrm{d}z \cdot \int g(y) \mathrm{d}y$$ – Willie Wong Jul 29 '13 at 8:36