Why we consider log likelihood instead of Likelihood in Gaussian Distribution

Question

I am reading Gaussian Distribution from a machine learning book. It states that -

We shall determine values for the unknown parameters $\mu$ and $\sigma^2$ in the Gaussian by maximizing the likelihood function. In practice, it is more convenient to maximize the log of the likelihood function. Because the logarithm is monotonically increasing function of its argument, maximization of the log of a function is equivalent to maximization of the function itself. Taking the log not only simplifies the subsequent mathematical analysis, but it also helps numerically because the product of a large number of small probabilities can easily underflow the numerical precision of the computer, and this is resolved by computing instead the sum of the log probabilities.

can anyone give me some intuition behind it with some example? Where the log likelihood is more convenient over likelihood. Please give me a practical example.

Thanks in advance!

Answer 1

1. It is extremely useful for example when you want to calculate the joint likelihood for a set of independent and identically distributed points. Assuming that you have your points: $$X=\{x_1,x_2,\ldots,x_N\} $$ The total likelihood is the product of the likelihood for each point, i.e.: $$p(X\mid\Theta)=\prod_{i=1}^Np(x_i\mid\Theta) $$ where $\Theta$ are the model parameters: vector of means $\mu$ and covariance matrix $\Sigma$. If you use the log-likelihood you will end up with sum instead of product: $$\ln p(X\mid\Theta)=\sum_{i=1}^N\ln p(x_i\mid\Theta) $$

2. Also in the case of Gaussian, it allows you to avoid computation of the exponential:

   $$p(x\mid\Theta) = \dfrac{1}{(\sqrt{2\pi})^d\sqrt{\det\Sigma}}e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}$$ Which becomes:

   $$\ln p(x\mid\Theta) = -\frac{d}{2}\ln(2\pi)-\frac{1}{2}\ln(\det \Sigma)-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)$$

3. Like you mentioned $\ln x$ is a monotonically increasing function, thus log-likelihoods have the same relations of order as the likelihoods:

   $$p(x\mid\Theta_1)>p(x\mid\Theta_2) \Leftrightarrow \ln p(x\mid\Theta_1)>\ln p(x\mid\Theta_2)$$

4. From a standpoint of computational complexity, you can imagine that first of all summing is less expensive than multiplication (although nowadays these are almost equal). But what is even more important, likelihoods would become very small and you will run out of your floating point precision very quickly, yielding an underflow. That's why it is way more convenient to use the logarithm of the likelihood. Simply try to calculate the likelihood by hand, using pocket calculator - almost impossible.

   Additionally in the classification framework you can simplify calculations even further. The relations of order will remain valid if you drop the division by $2$ and the $d\ln(2\pi)$ term. You can do that because these are class independent. Also, as one might notice if variance of both classes is the same ($\Sigma_1=\Sigma_2 $), then you can also remove the $\ln(\det \Sigma) $ term.

Answer 2

First of all as stated, the log is monotonically increasing so maximizing likelihood is equivalent to maximizing log likelihood. Furthermore, one can make use of $\ln(ab) = \ln(a) + \ln(b)$. Many equations simplify significantly because one gets sums where one had products before and now one can maximize simply by taking derivatives and setting equal to $0$.

Answer 3

Maybe a practical example for computer scientists: As @jojek mentioned

$$p(X|\Theta) = \prod_{i=1}^N p(x_i|\Theta)$$

and

$$\ln p(X| \Theta) = \sum_{i=1}^N \ln p(x_i| \Theta)$$

So if you execute a little python script you can directly see it:

>>> import numpy as np

# Create some small probability values.
>>> r = np.random.random_sample((100,)) * 0.000001

# Take the log of each (I ignore the logarithms basis, because it does not matter here).
>>> l = [np.log(x) for x in r]
# Compute product of probability values.
>>> np.product(r)
0.0

# Compute sum of log of probability values.
>>> np.sum(l)
-1472.245511811776

Here you can see that it is more practical to work with logarithms. Especially in machine learning where you compute with many very small numbers (e.g. check the weights of a neural network) this can lead to much better results.

Answer 4

Alright, first let's talk about what is likelihood. The likelihood is the probability of seeing certain data when the model is fixed (fixed means it is for a particular model or the model we have right now after training it for a particular number of epochs). Let's consider the model from a generative perspective. For any random model, it represents/summarizes some distribution of data with its parameter. Now you can think of the model as any sophisticated machine learning model like Neural Network with millions of parameters, or even a very simple model with 2 parameters (mean and standard deviation) which is in your case. All it does is summarizes some sample points and in this way, our target is to get an approximation of the population (as close as possible) from where the samples themselves are collected.

Now think you are training your model (in this case a Gaussian Distribution) to represent a particular set of data, such that you need not remember all thousands of data points, but you can capture its behavior with these two parameters (mean and standard deviation) of your model. Now, what should be our goal? To tune the parameter in such a way that it represents our current sample points, right? In other words, we want to tune it in such a way that, this model/distribution becomes as close as possible to the unknown distribution from which those points are even sampled. In other words, if we sample from our Gaussian model, it should be very likely to find points that are very close to the point on which this model is itself trained. But the question is how will you measure that your model is capable of producing points similar to its training data? This is where the MLE (Maximum Likelihood Estimation) comes.

Now, it is very logical that we are not interested to make the model learn a single point, but we want it to learn a set of points (training dataset). So, we have to measure the joint probability distribution of the whole dataset (condition on the model Θ, or we can say with respect to the model Θ). But now think, there are infinite possibilities of values, for an individual point the probability is very low (considered as 0 in continuous cases). Apart from this, the probability always lies in between [0, 1]. So, for each of the points, the probability is no more than 1. So, if you multiply them all together to find the joint probability (with respect to the model) then you are basically multiplying too many numbers which are no more than 1. Which will lead to a mathematical underflow in our computing machines. Note that, it is not a limitation of mathematics, but it is the limitation of the device where we will calculate it. Our computing devices work on a limited floating point precision. So, there is a very good chance, that multiplying such a small number will make the result 0. This is not what we want, right? Moreover, multiplying is a heavy operation. So, as a remedy, we take the log of the likelihood and try to maximize it, such that our model becomes more and more capable of imitating the training data.

As the log is a monotonically increasing function (that means, if you increase the value, the log of that value will also increase). So, as we just need to compare to find the best likelihood, we don't care what its actual value is, the only thing we care if the log-likelihood is increasing or not. Mathematically, the following is often computationally infeasible to perform: $$p(X\mid\Theta)=\prod_{i=1}^Np(x_i\mid\Theta)$$ So, we do the following: $$\ln p(X\mid\Theta)=\sum_{i=1}^N\ln p(x_i\mid\Theta)$$

What is Θ? It is nothing but your model (its parameters: mean and standard deviation).