I am studying a Tutorial on Maximum Likelihood Estimation in Linear Regression and I have a question.

When we have more than one regressor (a.k.a. multiple linear regression1), the model comes in its matrix form $y = X\beta + \epsilon$, (1)where $y$ is the response vector, $X$ is the design matrix with each its row specifying under what design or conditions the corresponding response is observed (hence the name), $\beta$ is the vector of regression coefficients, and $\epsilon$ is the residual vector distributing as a zero-mean multivariable Gaussian with a diagonal covariance matrix $\mathcal{N} \sim (0, \sigma^2I_N )$, where $I_N$ is the $N \times N$ identity matrix. Therefore $y \sim \mathcal{N}(X\beta,\sigma^2 I_N)$, (2)meaning that linear combination $X\beta$ explains (or predicts) response y with uncertainty characterized by a variance of $\sigma^2$.

Assume $y, \beta$, and $\epsilon \in \mathbb{R^n}$ Under the model assumptions, we aim to estimate the unknown parameters ($\beta$ and $\sigma^2$) from the data available ($X$ and $y$).

Maximum likelihood (ML) estimation is the most common estimator. We maximize the log-likelihood w.r.t. $\beta$ and $\sigma^2$ $\mathcal{L}(\beta,\sigma^2|y,X)=−\frac{N}{2}\log{2\pi}−\frac{N}{2}log{\sigma^2}− \frac{1}{2\sigma^2} (y−X\beta)^T(y−X\beta)$.

I am trying to understand that how the log-likelihood, $\mathcal{L}(\beta,\sigma^2|y,X)$, is formed. Normally, I saw these problems when we have $\mathbf{x_i}$ as vector of size d(d is number of parameter for each data). specifically, when $\mathbf{x_i}$ is a vector, I wrote is as $\ln \prod\limits_{i=1}^{N} \frac{1}{\sqrt { (2\pi)^d\boldsymbol \sigma^2 } } \exp\left(-{1 \over 2\sigma^2} (\mathbf{x_i}-\boldsymbol\mu)^{\rm T} \boldsymbol ({\mathbf x_i}-\boldsymbol\mu)\right) = \sum\limits{_{i}}\ln{\frac{1}{\sqrt { (2\pi)^d\boldsymbol \sigma^2 } } \exp\left(-{1 \over 2\sigma^2} (\mathbf{x_i}-\boldsymbol\mu)^{\rm T} \boldsymbol ({\mathbf x_i}-\boldsymbol\mu)\right)}$. But in the case that is shown in this tutorial, there is no index I to apply summation.

I would appreciate any insights on this problem. Thanks in advance.


I think it's relatively easy to get mixed up here due to notation. In the case you present from the textbook, they're considering a product of one-dimensional gaussians which are independent from each other, and then writing the form as a multi-dimensional gaussian (since then the covariance matrix of this multidimensional gaussian is exactly $\sigma^2I$). E.g. note that each sample has the form $y_i - (X\beta)_i \sim N(0, \sigma^2)$. Taking the product of these distributions yields the multi-dimensional gaussian above.

In your exposition, on the other hand, you're writing a multi-dimensional gaussian which is i.i.d.; this is different than what the textbook is referring to, since the mean should change between distributions (e.g. the samples are independent, but not identically drawn, since we're observing different data points with some additional noise, $\varepsilon \sim N(0, \sigma^2)$).

  • $\begingroup$ Thanks for your explanation. I am still confused. If they derive it by writing it as product what is the index they used? I mean I don't see any variable or index here. $\endgroup$ – Crimson Aug 24 '17 at 2:52
  • 1
    $\begingroup$ Let $f(x; \sigma^2)$ be the gaussian PDF, with mean zero and variance $\sigma^2$, evaluated at $x$ (with appropriate generalization to the multivariate case). Then we can write $\prod_i f(x_i\beta - y; \sigma^2) = f(X\beta - y; \sigma^2I)$. Perhaps it might help clear it up if you show that $\prod_i f(\mu_i; \sigma^2) = f(\mu; \sigma^2 I)$ for general $\mu_i$. $\endgroup$ – Guillermo Angeris Aug 24 '17 at 2:54
  • $\begingroup$ I did not see the point you mentioned before. Thanks again. $\endgroup$ – Crimson Aug 24 '17 at 3:04
  • 1
    $\begingroup$ Yes, I mean I did not see this step ,$\prod _i f(x_i\beta-y;\sigma^2) - f(X\beta - y ; \sigma^2I)$, before you mentioned. I got it now. Thanks! $\endgroup$ – Crimson Aug 24 '17 at 3:12
  • 1
    $\begingroup$ @Crimson Great! Glad to help! $\endgroup$ – Guillermo Angeris Aug 24 '17 at 3:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.