Can someone explain Dirac Distribution? I have been reading book on Deep Learning and in the chapter of probability and information theory I found this topic "Dirac Distribution and Empirical Distribution". It says:
In some cases, we wish to specify that all the mass in a probability distribution clusters around a single point.
I don't understand what this means? And also how this is related to machine learning, what is the significance of it?
I can understand a bit of mathematics of this but unable to picturize the concept, if someone can help me in this(in a more simpler way) it will be a real help.
PS: If you can provide some other study material regarding this it will be really helpful.
Thanks
 A: In Machine Learning, you use the $\delta$ probability distribution (concentrated at one point) when the true labels in a supervised learning problems are deterministic. In that case, the conditional probability of $\{Y|X=x\}$ is a delta centered at the true value $y=f(x)$. In contrast, when you do not have a deterministic label but different possible values for $y$ at a given $x$, your conditional distribution has some positive variance (spread).
A: The "mass" of a probability is just the numeric probability itself. In short, the excerpt is talking about a situation where a random variable $X$ takes a very narrow range of variables, or even takes one value almost surely. The latter case (with the value $0$) is called the $\delta$ distribution: It's the unique probability distribution with $P(\{0\}) = 1$.
There's a general theory of distributions, but here's the quick summary with the application of probability in mind. For a random variable $X$ with a "nice" continuous distribution, we can write
$$P(a < X < b) = \int_a^b dx\, f(x)$$
for some density function $f$. (I won't precisely define "nice" here, but see the Radon-Nikodym derivative for the details.) If we consider the reasonable random variable $X\equiv 0$, though, there's no suitable $f$; we'd have to have
\begin{align*}
\int_a^b\, dx\,f(x) = \begin{cases}
1 & \text{if $a < 0 < b$}; \\
0 & \text{otherwise},
\end{cases}
\end{align*}
but the integral on the left is continuous in $a$ and $b$. The solution is to posit a distribution--- not a function--- $\delta$ with the property that
\begin{align*}
\int_a^b dx\, f(x) \delta(x) = f(0)
\end{align*}
for any interval $[a, b]$ containing $0$. In order to make this rigorous, there are two similar ways of proceeding. The first is to choose a nice series of functions, say
\begin{align*}
\psi_n(x) &= \begin{cases}
n & \text{if $|x| < \frac{1}{2n}$}; \\
0 & \text{otherwise},
\end{cases}
\end{align*}
and take
\begin{align*}
\int_a^b dx\, f(x)\delta(x) = \lim_{n\to\infty} \int_a^b\, f(x)\psi_n(x),
\end{align*}
and similarly. It's then possible to work with $\delta$ formally and prove some useful results. 
The second method is to note that integration defines an inner product on "nice" functions:
$$\langle{f, g\rangle} = \int dx\, f(x)g(x).$$
(Take, for example, square-integrable functions on the real line; then the integral above converges by Hölder's inequality.) Normally, one would expect the Riesz representation theorem to hold, and every linear functional on the space of such $f$ is of the form $\langle{f, f_0\rangle}$ for some fixed $f_0$. In this case, however, it fails: $L(f) = f(0)$ is a perfectly nice functional, and it's not of that form for any $f_0$ by the same argument above. The workaround, then, is to just write $L(f) = \langle{f, \delta\rangle}$; the $\delta$ here is not actually a function, but simply a shorthand for $L$. As above, there are ways of extending this symbolism into a working calculus; we can even talk about things like $\delta'$, for example, by noting that $\langle{f', g\rangle} = \langle{f, g'\rangle}$ for any genuine functions $f, g$ by integration by parts, and extending the same conceit to $\delta$. (There are also ways of making this rigorous and precise. I'll also add parenthetically that although you mentioned machine learning, the main area where distributions pop up is in differential equations.)
