2
$\begingroup$

I'm reading the book "Mathematics for Machine Learning", it's a free book that you can find here. So I'm reading section 8.3 of the book which explains the maximum likelihood estimation (or MLE). This is my understanding of how MLE works in machine learning:

Say we have a dataset of vectors $(x_1, x_2, ..., x_n)$, we also have corresponding labels $(y_1, y_2, ..., y_n)$ which are real numbers, finally we have a model with parameters $\theta$. Now MLE is a way to find the best parameters $\theta$ for the model, so that model would map $x_n$ to $\hat{y}_n$ and $\hat{y}_n$ is as close to $y_n$ as possible.

For each $x_n$ and $y_n$ we have a probability distribution $p(y_n|x_n,\theta)$. Basically it estimates how likely our model with parameters $\theta$ will output $y_n$ when we feed it $x_n$ (and the bigger the probability the better).

We then take a logarithm of each of the estimated probabilities and sum up all the logarithms, like this: $$\sum_{n=1}^N\log{p(y_n|x_n,\theta)}$$

The bigger this sum the better our model with parameters $\theta$ explains the data, so we have to maximize the sum.

What I don't understand is how do we chose the probability distribution $p(y_n|x_n,\theta)$ and its parameters? In the book there is an Example 8.4, where they chose the probability distribution to be Gaussian distribution with zero mean, $\epsilon_n \sim \mathcal{N}(0,\,\sigma^{2})$. They then assume that the linear model $x_n^T\theta$ is used for prediction, so: $$p(y_n|x_n,\theta) = \mathcal{N}(y_n|x_n^T\theta,\,\sigma^{2})$$ and I don't understand why they replaced zero mean with $x_n^T\theta$, also where do we get covariance $\sigma^{2}$?

So this is my question, how do we chose the probability distribution and it's parameters? In the example above the distribution is Gaussian but it could be any other distribution from those that exist and different distributions have different types and numbers of parameters. Also as I understood each $x_n$ and $y_n$ have its own probability distribution $p(y_n|x_n,\theta)$ which even more complicates the problem.

I would really appreciate your help. Also note that I'm just learning the math for machine learning and not very skilled. If you need any additional info please ask in the comments.

Thanks!

$\endgroup$

2 Answers 2

0
$\begingroup$

enter image description here

$\epsilon_n\sim \mathcal N(0, \sigma^2)$ is not the probability distribution of the data, it is the probability of the random noise/ error.

They assume the data follow a perfectly linear relationship with noise given by $\epsilon _n$ for each data point (label pair) $(\textbf x_n, y_n) $, hence the equation of the least squares regression line $y=\textbf x^T\theta$ will have some noise to it. Putting these together the author gets that the $p(y_n|\textbf x_n, \theta, \sigma^2)=\mathcal N(\textbf x_n^T\theta, \sigma^2)$. You are right that the $\sigma^2$ is an extra parameter that would need to be estimated and should therefore be included in the conditional statement.

Each $p(y_n|\textbf x_n, \theta)$ (omitting the $\sigma^2)$ this time for brevity, has its own distribution but under the random sample assumption the labels are assumed to be independent from each other. Hence the overall likelihood is

$$\prod_{i=1}^n p(y_n|\textbf x_n,\theta)$$

due to their coming from a random sample. Taking the logarithm gives the log likelihood, in which you turn the product into a summation and move the logarithm inside:

$$\log \prod_{i=1}^n p(y_n|\textbf x_n,\theta)=\sum_{i=1}^n \log p(y_n|\textbf x_n,\theta)$$

$\endgroup$
8
  • 1
    $\begingroup$ Thanks a lot for your answer! However, I still have some questions: at first I thought that we assume that our data is generated by $x^T\theta$ and we use Gaussian distribution to measure how well our model performs but after I read your answer I'm starting to think that we assume that our data is generated by Gaussian distribution and what we're trying to do is to find the parameters of this distribution using MLE. Does that make sense? $\endgroup$
    – user950315
    Jul 21, 2021 at 17:34
  • $\begingroup$ Also I don't understand why did they set the mean of the Gaussian distribution to $x_n^T\theta$? $\endgroup$
    – user950315
    Jul 21, 2021 at 17:37
  • 1
    $\begingroup$ @acmpo6ou The way they got that the data is normally distributed is by specifying that they follow a linear model $y_n=x_n^T\theta=\theta_0+x_{n1}\theta_1+\dots+x_{nm}\theta_m$. Then they assumed that the data followed Gaussian noise $\mathcal N(0, \sigma^2)$. Putting these together, the data no longer follow an exact line, but are randomly spread about the line in n-dimensional space and are $\mathcal N(x_n^T\theta, \sigma^2)$. Which is to say, there are as many normal distributions as there are data points in a sense. This works if you have paired data $(y_i, \textbf x_i)$. $\endgroup$
    – Vons
    Jul 21, 2021 at 18:11
  • 1
    $\begingroup$ I assume there are other machine learning techniques than this, such as random forest, which are nonparametric. This is also different from saying singular observations ($x_i$) that might come from say an exponential distribution. $\endgroup$
    – Vons
    Jul 21, 2021 at 18:12
  • 1
    $\begingroup$ @acmpo6ou yes, MLE can be used to estimate two parameters. $\endgroup$
    – Vons
    Jul 22, 2021 at 14:22
0
$\begingroup$

In addition to @Jellyfish's answer, sometimes we just use Gaussian (or some other particular) distribution in MLE even if we know it may not be the true distribution of data, this method is called Quasi-MLE. In application of machine learning, the dataset is often large in sample size and maybe in dimension. This feature allows us to focus mainly on the asymptotic properties of estimators, like consistency. MLE can still be consistent even in the case that the probability distribution is mis-specified. After all, only god knows what the true distribution is.

$\endgroup$
3
  • $\begingroup$ Thanks a lot! So do you mean that we can try different distributions and chose the one that works best? $\endgroup$
    – user950315
    Jul 21, 2021 at 17:42
  • 1
    $\begingroup$ @acmpo6ou Sure we can use different distributions. The point is, the consistency of MLE ensures that even if the distribution function we use is mis-specified, as long as the dataset is large enough, the estimation is still acceptable. $\endgroup$
    – Q9y5
    Jul 22, 2021 at 0:05
  • $\begingroup$ Also from the wikipedia link that you gave I found out that $\sigma^2$ is a typical nuisance parameter meaning that it's not of a primary interest in gaussian distribution because we are interested more in the mean. That's why it's probably not as important to estimate in the case of MLE. $\endgroup$
    – user950315
    Jul 22, 2021 at 6:12

You must log in to answer this question.