If I have this linear regression equation:

$$y=X\beta+\epsilon $$

($x$ and $\beta$ are vectors)

The likelihood function can be written as

$$L= \prod_{n=1}^N N(y_n ;x_n ,\beta ,\sigma^2)=(2\pi \sigma^2)^{-\frac {N}{2}}\exp\left\{ \frac{-1}{2\sigma^2} (y- X\beta)'(y-X\beta)\right\}$$

However, what changes if the $\epsilon$ term is independent but non-identically normally distributed? I know it would mean that the mean is the same but variance of the noise changes.

How do I represent it in the likelihood function to proceed with MLE of $\beta$?

  • Do I simply take the sum of all variances in the term above? Like $\frac{\sum\sigma^2_n}{N}$?
  • $\begingroup$ Your likelhood function is missing $\exp$, as in $\exp\left\{ \vphantom{\dfrac 1 1} \cdots\cdots \right\}$. ${}\qquad{}$ $\endgroup$ – Michael Hardy Sep 25 '14 at 22:45
  • $\begingroup$ You're using both capital $X$ and lower-case $x$, in the two expressions $X\beta$ and $x^T\beta$. Normally I'd expect the same matrix in both places, which would usually have more than one column. ${}\qquad{}$ $\endgroup$ – Michael Hardy Sep 25 '14 at 22:47
  • $\begingroup$ Gracias! Again. $\endgroup$ – Raaj Sep 25 '14 at 22:52
  • $\begingroup$ Independent but non-identically normally distributed errors is often called heteroscedasticity, for literature search. $\endgroup$ – Joe Sep 25 '14 at 22:52
  • $\begingroup$ I just found the term on Wiki! :D Thank you. If you have any resources you prefer to read on this further, please do let me know. I'd be grateful. $\endgroup$ – Raaj Sep 25 '14 at 22:54

The generalization is as follows:

$$ L = \frac{1}{\sqrt{(2\pi)^n \left|\Sigma \right|}} \exp \left\{ -\frac{1}{2}(y-X\beta)^T \Sigma^{-1}(y-X\beta)\right\} $$

where $\Sigma$ is the covariance matrix of the multivariate Gaussian distribution, and $|\cdot|$ denotes its determinant. If the noises are independent, then it will be a diagonal matrix with noise variances on the diagonal.

  • $\begingroup$ Quick question - Why is there an inverse of co-variance matrix in the term inside exponential? Independence of course means that there is no correlation and hence the diagonal matrix makes sense. $\endgroup$ – Raaj Sep 25 '14 at 23:02
  • 1
    $\begingroup$ In the scalar case, the variance is in the denominator, right? Think of it as a generalization of that. In particular, for a diagonal matrix, the inverse matrix is simply another diagonal matrix with each term on the diagonal is replaced with its multiplicative inverse. $\endgroup$ – karakusc Sep 25 '14 at 23:08
  • $\begingroup$ How much estimation you can do with this might depend on whether there is more than one $y$-value for each $x$-value. If, for example, one knows that the variances of two $y$ observations the same if the $x$ observation for both is the same, then one could estimate that variance based on the data. But estimating a variance based on a single observation is problematic, to say the least. $\endgroup$ – Michael Hardy Sep 25 '14 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.