When describing the Dirac delta function and Dirac distribution, my textbook says the following:

In some cases, we wish to specify that all the mass in a probability distribution clusters around a single point. This can be accomplished by defining a PDF using the Dirac delta function, $\delta(x)$:

$$p(x) = \delta(x - \mu)$$

The Dirac delta function is defined such that it is zero valued everywhere except $0$, yet integrates to $1$. The Dirac delta function is not an ordinary function that associates each value $x$ with a real-valued output; instead it is a different kind of mathematical object called a generalized function that is defined in terms of its properties when integrated. We can think of the Dirac delta function as being the limit point of a series of functions that put less and less mass on all points other than zero.

By defining $p(x)$ to be $\delta$ shifted by $- \mu$ we obtain an infinitely narrow and infinitely high peak of probability mass where $x = \mu$.

Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron; Bach, Francis. Deep Learning (NONE) (Page 64). The MIT Press. Kindle Edition.

  1. I'm not sure that I understand what is meant by this part:

We can think of the Dirac delta function as being the limit point of a series of functions that put less and less mass on all points other than zero.

If someone could please demonstrate what this is (mathematically), then I'd greatly appreciate it.

  1. If we obtain an infinitely narrow and infinitely high peak, as shown in this Wikipedia article, then wouldn't the integral over that peak be equal to $0$, since the peak is infinitely narrow and therefore has no area underneath it? This is my understanding from my study of the Riemann integral.

Thanks for any clarification.

  • 1
    $\begingroup$ This animation shows what it means: en.wikipedia.org/wiki/Dirac_delta_function#/media/… $\endgroup$ – md2perpe Sep 3 '18 at 6:05
  • $\begingroup$ This section of the same Wikipedia page goes into a bit more detail regarding the weak sense in which the Dirac delta function can be characterized as a limit of a sequence of functions. $\endgroup$ – Bungo Sep 3 '18 at 6:09
  • $\begingroup$ @Bungo I'm looking for a more elaborate, elementary, and intuitive explanation than what is provided in the Wikipedia article. I'm not experienced in analysis, so it's difficult for me to appreciate the article's explanation. $\endgroup$ – The Pointer Sep 3 '18 at 6:19
  • 2
    $\begingroup$ @ThePointer: One thing to understand here is that "limit point of a series of functions" doesn't mean what it sounds like it means. It means "first reinterpret the functions as generalized functions, and then take the limit point of the series of generalized functions". $\endgroup$ – Hurkyl Sep 3 '18 at 6:27
  • 1
    $\begingroup$ ... and another thing to understand that integration when generalized functions are involved is defined via the distribution and nothing to do with ordinary integrals. (unless the distribution itself is defined to give a value related to an ordinary integral) $\endgroup$ – Hurkyl Sep 3 '18 at 6:29

The idea is that the "dirac delta function" at $0,$ denoted by $\delta_0,$ is not an ordinary function. But somehow we can still integrate with it. For every continuous $g$ we have

$$\int_{\mathbb R} g\,\delta_0 = g(0).$$

However, no integrable function could have this property. That is, if $f$ is integrable on $[-a,a],$ then

$$\int_{-a}^a g(x) f(x)\,dx = g(0)$$

will fail for some continuous $g.$

On the other hand, and staying with elementary means, we can find a sequence $f_n$ of continuous functions on $\mathbb R$ such that

$$\lim_{n\to \infty}\int_{\mathbb R} g(x) f_n(x)\,dx = g(0)$$

for every continuous $g$ on $\mathbb R .$

Proof: Define $f_n$ on $[-1/n,1/n]$ to have the triangular graph that joins the points $(-1/n,0), (0,n), (1/n,0);$ define $f_n=0$ everywhere else. You can see that the $f_n$'s live in ever smaller intervals centered at $0,$ but nevertheless $\int_{\mathbb R} f_n = 1$ for every $n.$

Let $g$ be continuous. Then for each $n$ there exists $c_n\in [-1/n,1/n]$ such that $|g(c_n)-g(0)|$ is the maximum of $|g(x)-g(0)|$ on the interval $[-1/n,1/n].$ Thus

$$|\int_{\mathbb R} g(x) f_n(x)\,dx - g(0)| = |\int_{\mathbb R} [g(x)-g(0)] f_n(x)\,dx|$$ $$ \le |g(c_n)-g(0)| \int_{\mathbb R} f_n = |g(c_n)-g(0)|\cdot 1.$$

As $n\to \infty,$ $c_n\to 0,$ so the last expression $\to 0$ by the continuity of $g$ at $0.$

  • $\begingroup$ Thanks for the answer. I've understood everything so far, but I don't understand the part where you wrote "... nevertheless $\int_{\mathbb R} f_n = 1$ for every $n$". I don't see how this is the case? For instance, if $n = 3$, then we have $\int_{-1/3}^{1/3} f_n$, and how do we know that this is equal to $1$? This part of the concept is something I'm not understanding. Thanks. $\endgroup$ – The Pointer Sep 16 '18 at 7:00
  • 1
    $\begingroup$ @ThePointer he's defining the $f_n$'s to have integral 1. The area of a triangle is 1/2 base times height. The size of base is 2/n, so the height he's making the triangles is $n$. $\endgroup$ – mathworker21 Sep 16 '18 at 7:50
  • $\begingroup$ @mathworker21 ahh, ok, that makes sense. Thanks! The next part that I'm confused about is the algebra in $|\int_{\mathbb R} g(x) f_n(x)\,dx - g(0)| = |\int_{\mathbb R} [g(x)-g(0)] f_n(x)\,dx|$. On the left-hand side, $dx$ was not applied to $g(0)$ -- only $g(x)f_n(x)$; but on the right-hand side, it is also applied to $g(0)$? Also, on the right-hand side, $f_n(x)$ is somehow factored out of $g(x) - g(0)$, even though we never had $g(0)f_n(x)$ on the left-hand side? $\endgroup$ – The Pointer Sep 16 '18 at 8:04
  • 1
    $\begingroup$ @ThePointer $\int f_n(x)dx = 1 \implies \int g(0) f_n(x)dx = g(0)$ $\endgroup$ – mathworker21 Sep 16 '18 at 9:50
  • $\begingroup$ @mathworker21 Ok, I understand now. Thank you for the assistance! $\endgroup$ – The Pointer Sep 16 '18 at 11:15

Suppose we had a sequence of functions \begin{equation} f_n(x) = \cases{ n \hspace{2mm}\textrm{if}\hspace{2mm} x\in [0, \frac{1}{n}]\\ 0\hspace{2mm}\textrm{if}\hspace{2mm} x\notin [0,\frac{1}{n}]} \end{equation}

Suppose we had another function $g(x)$ and we integrate $\int_{-\infty}^\infty f_n(x)g(x)dx$.

We are going to obtain:

\begin{equation} \int_{-\infty}^\infty f_n(x)g(x)dx = nG(x)|_{0}^{\frac{1}{n}} \end{equation}

Evaluating the integral at the two values gives:

\begin{equation} \begin{split} nG(x)|_{0}^{\frac{1}{n}} & = n(G(\frac{1}{n}) - G(0))\\ & = n(G(0 +\frac{1}{n}) - G(0))\\ & = \frac{G(0 +\frac{1}{n}) - G(0)}{\frac{1}{n}} \end{split} \end{equation}

Taking the limit as $n\rightarrow \infty$ gives us:

\begin{equation} \begin{split} \frac{G(0 +\frac{1}{n}) - G(0)}{\frac{1}{n}} & \approx\frac{G(0 +\epsilon) - G(0)}{\epsilon} \end{split} \end{equation}

Which is the derivative of $G(x)$ evaluated at $0$. The derivative of $G(x)$ is $g(x)$, so:

\begin{equation} \lim_{n\rightarrow\infty}\int_{-\infty}^\infty f_n(x)g(x)dx = g(0) \end{equation}

  • $\begingroup$ Thanks for the answer. My concern is that this doesn't really explain anything to me to clarify my questions, so it's not very enlightening. Also, I don't understand how $\lim_{n \to \infty} \dfrac{1}{n} = \epsilon$; as I understand it, $\lim_{n \to \infty} \dfrac{1}{n} = 0$. $\endgroup$ – The Pointer Sep 15 '18 at 6:27
  • 2
    $\begingroup$ @ThePointer $\epsilon$ is commonly used to mean "small number," so here it is the same as $\frac1n$; in essence $\lim_{n\to\infty} \frac{1}{n} = \lim_{\epsilon \to 0} \epsilon$. $\endgroup$ – Dwagg Sep 15 '18 at 21:24
  • $\begingroup$ Ok, thanks. But it's still not clear to me how this is addressing questions 1 and 2. Can you please elaborate and clarify? $\endgroup$ – The Pointer Sep 16 '18 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.