Overcoming Linear Regression Assumptions

I'm a beginner in econometrics (learning on my own, and not from school) and I'm trying to build an intuition to understanding linear regression. We know that modeling real world data is bound to break many of the rules surrounding linear regression. Does it mean breaking any of those rules would render the use of linear regression ineffective? Or should we test the materiality of breaking those rules. How do we go about testing the materiality? Otherwise, what alternatives should we use?

The assumptions underlying linear regression states that:

1. Expected value of the error term, conditional on the independent variable, is zero. Qn: What happens if it's not zero, how do you test for it, and when is it considered significant?

2. All x, y observations are i.i.d. Qn: How would that affect my regression results if they weren't?

3. It is unlikely that large outliers will be observed in the data. Qn: What should you do if large outliers are observed? should you ignore those data points? if yes, how would you know those data points can be ignored?

4. Independent variable is uncorrelated with the error terms Qn: How do you test for this correlation, and are there ways to mitigate this correlation?

5. Homoskedasticity Qn: what if the variance of the error term is not constant? What should you do?

-
All of these issues have been adressed in the literature, but the work is usually mch more complicated than linear regression under standard assumptions. A good, but probably still too advanced, starting point would be reading "Asymptotic Theory for Econometricians" by Halbert White. – Michael Greinecker Jul 2 '12 at 13:52
I'm pretty sure a good portion of your questions have been asked and addressed at CV; for 3. in particular, it is now considered poor form to delete outliers. Either use "robust methods", or better yet, look into why you're getting outliers in the first place. – J. M. Jul 5 '12 at 6:14

I have had numerous discussions about this on CrossValidated. Linear regression is a very useful tool. The assumptions made when doing modeling in practice never hold exactly. George Box is known for say "All models are wrong. but some are useful." The key assumptions for applying OLS regression are 1. error component of the model are iid normal with 0 mean and constant variance 2. The covariates X are measured without error

Assumptions in 1 can be tested on the computed model residuals. Look at qxq plot of the residual or perform a goodness of fit test for normality such as the Shapiro-Wilk test. You can apply a t ets on the residuals to test for zero mean. If you plot the residuals against the covariates you can see visually if the variance changes with the covariate and if the mean departs from 0. Slight departures from normality will not hurt the model. But large departures and outliers will. If the assumptions are severely violated you can use a robust regression procedure. Robust regression methods use a different loss function that does not penalize so much for large individual errors. Least squares puts a high penalty on large errors and thus forces the curve to fit the outliers too well. Robust procedures like MAD do not.

Some problems with the residuals can be due to (1) missing covariates or (2) nonlinearity in the parameters. These can be remedied by introducing additional covariates in (1) or applying a nonlinear regression model in 2.

-
could you provide me this "site" which you mentioned above? – louis xie Jul 5 '12 at 6:10
Perhaps Michael is referring to the site to which J. M. has linked in his comment on the original question. – Gerry Myerson Jul 5 '12 at 6:19
Yes I am very active on CV and have discussed this there. – Michael Chernick Jul 5 '12 at 11:18
Are there any specific links I could refer to? I do not have a formal training in this area, and referring me to that site is like finding a pin in the forest, and I could hardly understand even the topic titles themselves :( – louis xie Jul 12 '12 at 3:38

You would want to start with the regression model when the explanatory variable $X$ is fixed rather than $\it{random}$; you can change the values of experimental setting, for example, you can test the effects of different fertilizer, the amount of water etc on the growth of some plant. Then, 1,2,4 and 5 are compromized as reasonalbe assumptions, and when 3 happens you may discard the outcome and try the identical setting because there may have been something strange happening you weren't aware of in the experiment. The linearity assumption of the conditional mean of $y$ is a simply assumption, but mathematically convenient. For example, the expectation operator is basically integration and liner operatot. So if the model is linear, it is easier to handle. I also got the impression that you may not have actually grasped those assumptions solidly from how you stated. The first assumption is called "strong exogenity" assumption, stated as $E(\epsilon|X)=0$. It means that a function $\mu(X)=E(\epsilon|X)$ does not actually depend on $X$. As I mentioned, the expecation operator is an integration which integrates out the variable $\epsilon$ , so $E(\epsilon|X)$ is a function of $X$. Therefore, how I stated for the strict exogeneity assumption actually implies that $E(\epsilon|X)$ just has to be simpley constant, does not have to be zero.

-