# Linear Regression with independent but non-identical noise

If I have this linear regression equation:

$$y=X\beta+\epsilon$$

($x$ and $\beta$ are vectors)

The likelihood function can be written as

$$L= \prod_{n=1}^N N(y_n ;x_n ,\beta ,\sigma^2)=(2\pi \sigma^2)^{-\frac {N}{2}}\exp\left\{ \frac{-1}{2\sigma^2} (y- X\beta)'(y-X\beta)\right\}$$

However, what changes if the $\epsilon$ term is independent but non-identically normally distributed? I know it would mean that the mean is the same but variance of the noise changes.

How do I represent it in the likelihood function to proceed with MLE of $\beta$?

• Do I simply take the sum of all variances in the term above? Like $\frac{\sum\sigma^2_n}{N}$?
• Your likelhood function is missing $\exp$, as in $\exp\left\{ \vphantom{\dfrac 1 1} \cdots\cdots \right\}$. ${}\qquad{}$ – Michael Hardy Sep 25 '14 at 22:45
• You're using both capital $X$ and lower-case $x$, in the two expressions $X\beta$ and $x^T\beta$. Normally I'd expect the same matrix in both places, which would usually have more than one column. ${}\qquad{}$ – Michael Hardy Sep 25 '14 at 22:47
• Gracias! Again. – Raaj Sep 25 '14 at 22:52
• Independent but non-identically normally distributed errors is often called heteroscedasticity, for literature search. – Joe Sep 25 '14 at 22:52
• I just found the term on Wiki! :D Thank you. If you have any resources you prefer to read on this further, please do let me know. I'd be grateful. – Raaj Sep 25 '14 at 22:54

$$L = \frac{1}{\sqrt{(2\pi)^n \left|\Sigma \right|}} \exp \left\{ -\frac{1}{2}(y-X\beta)^T \Sigma^{-1}(y-X\beta)\right\}$$
where $\Sigma$ is the covariance matrix of the multivariate Gaussian distribution, and $|\cdot|$ denotes its determinant. If the noises are independent, then it will be a diagonal matrix with noise variances on the diagonal.
• How much estimation you can do with this might depend on whether there is more than one $y$-value for each $x$-value. If, for example, one knows that the variances of two $y$ observations the same if the $x$ observation for both is the same, then one could estimate that variance based on the data. But estimating a variance based on a single observation is problematic, to say the least. – Michael Hardy Sep 25 '14 at 23:11