1
$\begingroup$

I am studying a Tutorial on Maximum Likelihood Estimation in Linear Regression and I have a question.

When we have more than one regressor (a.k.a. multiple linear regression1), the model comes in its matrix form $y = X\beta + \epsilon$, (1)where $y$ is the response vector, $X$ is the design matrix with each its row specifying under what design or conditions the corresponding response is observed (hence the name), $\beta$ is the vector of regression coefficients, and $\epsilon$ is the residual vector distributing as a zero-mean multivariable Gaussian with a diagonal covariance matrix $\mathcal{N} \sim (0, \sigma^2I_N )$, where $I_N$ is the $N \times N$ identity matrix. Therefore $y \sim \mathcal{N}(X\beta,\sigma^2 I_N)$, (2)meaning that linear combination $X\beta$ explains (or predicts) response y with uncertainty characterized by a variance of $\sigma^2$.

Assume $y, \beta$, and $\epsilon \in \mathbb{R^n}$ Under the model assumptions, we aim to estimate the unknown parameters ($\beta$ and $\sigma^2$) from the data available ($X$ and $y$).

Maximum likelihood (ML) estimation is the most common estimator. We maximize the log-likelihood w.r.t. $\beta$ and $\sigma^2$ $\mathcal{L}(\beta,\sigma^2|y,X)=−\frac{N}{2}\log{2\pi}−\frac{N}{2}log{\sigma^2}− \frac{1}{2\sigma^2} (y−X\beta)^T(y−X\beta)$.

I am trying to understand that how the log-likelihood, $\mathcal{L}(\beta,\sigma^2|y,X)$, is formed. Normally, I saw these problems when we have $\mathbf{x_i}$ as vector of size d(d is number of parameter for each data). specifically, when $\mathbf{x_i}$ is a vector, I wrote is as $\ln \prod\limits_{i=1}^{N} \frac{1}{\sqrt { (2\pi)^d\boldsymbol \sigma^2 } } \exp\left(-{1 \over 2\sigma^2} (\mathbf{x_i}-\boldsymbol\mu)^{\rm T} \boldsymbol ({\mathbf x_i}-\boldsymbol\mu)\right) = \sum\limits{_{i}}\ln{\frac{1}{\sqrt { (2\pi)^d\boldsymbol \sigma^2 } } \exp\left(-{1 \over 2\sigma^2} (\mathbf{x_i}-\boldsymbol\mu)^{\rm T} \boldsymbol ({\mathbf x_i}-\boldsymbol\mu)\right)}$. But in the case that is shown in this tutorial, there is no index I to apply summation.

I would appreciate any insights on this problem. Thanks in advance.

$\endgroup$

2 Answers 2

1
$\begingroup$

I think it's relatively easy to get mixed up here due to notation. In the case you present from the textbook, they're considering a product of one-dimensional gaussians which are independent from each other, and then writing the form as a multi-dimensional gaussian (since then the covariance matrix of this multidimensional gaussian is exactly $\sigma^2I$). E.g. note that each sample has the form $y_i - (X\beta)_i \sim N(0, \sigma^2)$. Taking the product of these distributions yields the multi-dimensional gaussian above.

In your exposition, on the other hand, you're writing a multi-dimensional gaussian which is i.i.d.; this is different than what the textbook is referring to, since the mean should change between distributions (e.g. the samples are independent, but not identically drawn, since we're observing different data points with some additional noise, $\varepsilon \sim N(0, \sigma^2)$).

$\endgroup$
6
  • $\begingroup$ Thanks for your explanation. I am still confused. If they derive it by writing it as product what is the index they used? I mean I don't see any variable or index here. $\endgroup$
    – Crimson
    Aug 24, 2017 at 2:52
  • 1
    $\begingroup$ Let $f(x; \sigma^2)$ be the gaussian PDF, with mean zero and variance $\sigma^2$, evaluated at $x$ (with appropriate generalization to the multivariate case). Then we can write $\prod_i f(x_i\beta - y; \sigma^2) = f(X\beta - y; \sigma^2I)$. Perhaps it might help clear it up if you show that $\prod_i f(\mu_i; \sigma^2) = f(\mu; \sigma^2 I)$ for general $\mu_i$. $\endgroup$ Aug 24, 2017 at 2:54
  • $\begingroup$ I did not see the point you mentioned before. Thanks again. $\endgroup$
    – Crimson
    Aug 24, 2017 at 3:04
  • 1
    $\begingroup$ Yes, I mean I did not see this step ,$\prod _i f(x_i\beta-y;\sigma^2) - f(X\beta - y ; \sigma^2I)$, before you mentioned. I got it now. Thanks! $\endgroup$
    – Crimson
    Aug 24, 2017 at 3:12
  • 1
    $\begingroup$ @Crimson Great! Glad to help! $\endgroup$ Aug 24, 2017 at 3:16
0
$\begingroup$

It is the same, only in matrix notation.

First realize that your "$X$" is now $Y$ your outcome variable. Now $X$ are the regressors. Next realize that $\mu$ is the regression model, i.e. $X\beta$, as you assume this is the expected value of the model. Next realize, that $Y$ is an $(n,1)$ vector with $n$ observations, $X$ is $(n,d)$ matrix, with $d$ parameters, and $\beta$ is $(d,1)$ vector. Also, by the model assumptions $\Sigma = \sigma^2I \;,\;\Sigma^{-1}=\sigma^{-2}I$.

So the likelihood function is:

$$ \mathcal{L}(\beta, \sigma ^2 | Y, X) = (\frac{1}{\sqrt{2\pi}\sigma})^ne^{-\frac{1}{2\sigma^2}(Y-X\beta)^T(Y-X\beta)} $$

And if you take the log of it, you get what's in the text book.

You don't need a sum in matrix notation. Notice that $(Y-X\beta)^T(Y-X\beta)$ is a scalar, which contains the sum you want, i.e. it is equal to $\sum_i^n(y_i-X\beta_i)^2 =\sum_i^n(y_i-\sum_j^d(x_{ij}\beta_j))^2$.

Next you can differentiate by $\beta$ or $\sigma$ to find their MLE.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .