In Linear Regression models, given observed variables $x_1, x_2, x_3, ..., x_k$, unobserved (or predicted) variable $y$, and model parameters $\beta_0, \beta_1, \beta_2, \beta_3, ..., \beta_k$, it can be written as $$ y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k + \epsilon $$ where, $\epsilon$ is the noise term.

In vector notation the same thing can be written down as: $$ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \epsilon $$

Now this can be solved using maximum-likelihood estimation over $\beta$. That is: $$ \hat{\beta} = argmax_{\beta} \Pr(\mathbf{y}|\mathbf{X}, \boldsymbol{\beta}) $$

Now I read somewhere that

In order to specify $\Pr(\mathbf{y}|\mathbf{X}, \boldsymbol{\beta})$ mathematically, we need to make assumptions about the noise term $\epsilon$. A common assumption is that $\epsilon$ follows a Gaussian distribution with zero mean and variance $\sigma_{\epsilon}^{2}$, $$ \epsilon \sim N(0, \sigma_{\epsilon}^{2}) $$ This implies that the conditional probability density function of the output $Y$ for a given value of the input $X = x$ is given by $$ \Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2}) $$

Now my question is, what has the distribution of $\epsilon$ (Normal distribution is this case), got to do with the distribution of the MLE?

In other words, why $\epsilon \sim N(0, \sigma_{\epsilon}^{2})$, also implies $$ \Pr(y|x, \beta) = N(y | \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k, \sigma_{\epsilon}^{2}) $$

Thank you in advance.


1 Answer 1


Note that if $\epsilon_i \sim_{iid} \mathcal{N}(0,\sigma^2)$, then $Y_i|\mathrm{x}_i$ is a linear combination of a normal random variable. Namely, as given values, and $y_i = f(x_1,..,x_p)+\epsilon_i$, then $y_i$ is linear combination of normal r.v., thus is itself normal with the following parameters,
$$ \mathbb{E}[y_i|x_1,...,x_p]= f(x_1,...,x_p)+\mathbb{E}[\epsilon_i|x_1,..,x_p]=f(x_1,...,x_p), $$
and $$ \operatorname{Var}[y_i|x_1,...,x_p]= 0+\operatorname{Var}[\epsilon_i|x_1,..,x_p]=\sigma^2. $$
When in you case $f$ is linear function of the form $\beta_0 + \sum_{j=1}^pX_j$.

  • $\begingroup$ Thank you for the explanation :) $\endgroup$
    – Koustav
    Feb 15, 2018 at 9:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .