In one of my recent statistics courses, our teacher introduced the linear regression model. The typical $y=\alpha + \beta X + \epsilon$, where $\epsilon$ is a "random" error term. The teacher then proceeded to explain that this error term is normally distributed and has a mean zero.

The error term is what is confusing me. What exactly does random mean? My back ground in statistics is very low level, but I understand that a random variable is defined as a mapping from a sample space to the real numbers. This definition makes sense, but the assumption of a zero mean is what I get tripped up on. How can we assume this fact?

I've been trying to think about it intuitively, and can only think that in regards to the real numbers, zero is in a sense "the middle ground" and splits up the reals into 2 "equal length" parts. However, I know that the reals are uncountable so this may be a case where my intuition is incorrect.

I apologize in advance if my question is confusing.

  • 1
    The fact that you're in (what I assume) is an undergrad-level stats course and know what a mapping is is quite fascinating, I have to admit. [I tutored statistics for four years in my undergrad.] – Clarinetist Dec 4 '14 at 19:08
up vote 5 down vote accepted

Here's the general idea - someone who has a better background than I do in statistics could probably give a better explanation. So you have this linear regression model: $$Y = \alpha + \beta X + \epsilon $$ where $\epsilon$ follows a normal distribution with mean $0$.

What exactly does random mean? My back ground in statistics is very low level, but I understand that a random variable is defined as a mapping from a sample space to the real numbers. This definition makes sense, but the assumption of a zero mean is what I get tripped up on. How can we assume this fact?

Personally, I've always taken the idea that $\epsilon$ follows a normal distribution with mean $0$ as an axiom of sorts for the linear regression model. My understanding is that it's just something nice we would like the linear regression model to have and lends itself well to certain properties. Remember:

Essentially, all models are wrong, but some are useful.

which is attributed to George E.P. Box.

Why would we want such an axiom? Well... on average, it would be nice to have zero error.

In my honest opinion (this is based off the little measure-theoretic probability I have studied), it would be best to approach this idea of "randomness" intuitively, as you would in an undergraduate probability course.

The idea about anything that is random is that you will never know the value of it. So, in an undergraduate probability class, what you do is you assign probabilities to the values your quality of interest can take by creating a probabilistic model. Your model, 99% of the time, won't be perfect, but that doesn't stop anyone from not trying.

The normal distribution with mean 0 is just an example of a probabilistic model that statisticians feel is a suitable model for the error term. It isn't perfect, but it's suitable for most purposes. I worked with a professor whose focus is on assuming a skew-normal error term, which complicates things, but is usually more realistic, since, in reality, not everything looks like a bell curve.

My two cents. Hopefully I've helped somewhat.

  • If there are two observations, and therefore two error terms, and the first error is -5, and the second is 5, then the mean would be zero - would this be an example of "having a mean zero"? Thank you. – Sabuncu Jan 18 '16 at 18:23

Basically, the errors represent everything that the model does not have into account. And why is that? Because it would be extremely unlikely for a model to perfectly predict a variable, as it is impossible to control every possible condition that may interfere with the response variable. The errors may also include reading or measuring inaccuracies. Considering the regression line of best fit, the errors are based on the distance from each point to that line.

The Central Limit Theorem is behind the assumption of the errors following a normal distribution. It states that the distribution of the sum of a large number of random variables will tend towards a normal distribution. And actually, in the real world, the majority of the observable errors appear to be distributed that way; which helps us to extrapolate to the unobservable errors.

Another assumption made is that each data point has its own independent associated error, i.e., the errors are independent from one another, which helps us assume they occur randomly.

And because the errors occur randomly, it is expected each data point has equal probability of appearing above or bellow the line of best fit created by the regression (positive error values for the data points with a higher value than the one predicted by the line, and negative error values for the data points with a smaller value predicted by the line), meaning if you summed up every error it would result in a value very close to zero.

Hope to have helped.

The assumption of mean 0 is a normalization that must be made because you already have a constant term in the regression. It relates to the issue of identification - that you as the researcher cannot tell the difference between the constant term in the regression and the mean of the error term.

Proof: Suppose that $\epsilon$ is not mean 0

Let $\bar{\epsilon}$ denote the mean of $\epsilon$. Then I can re-write your model as

$Y = (\alpha + \bar{\epsilon}) + \beta X + (\epsilon - \bar{\epsilon})$.

let $\tilde{\alpha} = \alpha + \bar{\epsilon} $ and $\tilde{\epsilon} = \alpha + \bar{\epsilon}$

-->$Y = \tilde{\alpha}+ \beta X + \tilde{\epsilon} $.

This model is identical to yours except it now has a mean-zero error term and the new constant combines the old constant and the mean of the original error term.

Marcus Dupree,

I was reading the book "Introduction to Statistics" by Trevor Hastie and Robert Tibshirani. The second chapter in that book deals with regression models. Authors, when explaining the relationship between Y and Xs, they mentioned about existence of "Random error". They further quoted that the "random error is independent of X and has mean zero". I had the question you had.

Why random error and why it's mean should be 0?

Then I tried to understand it by some logic, also google searches.
what I found is this,

Ideally, we are trying to represent an one to one relationship between Target Variable and Independent Variables. Assuming that we have collected all of the independent variables that are required to explain Y, then the relationship can be represented in a form of a function Y = f(X) ( f(X) will exactly explain Y ).

What are the chances that our assumptions are right with anyone observation is picked at random and apply the function?

if 'i is the n'th observation, applying Yi = f(Xi), you will see the difference between Actual Yi and f(Xi), against our assumption Yi = f(Xi). This difference is called error. Backing our initial assumption, 'Y is completely explained by all Xs we have', we assume this error as a random or Noise and still stick strongly to our initial assumption.

Applying the function f(X) to specific observation may result in Random Error but on the whole, according to our initial assumption, there will not be any error. How is that possible? Specific Observations error, on the whole, no error!

if I have 10 observations, I may have the maximum of 10 errors and they are random. if y is target and 'e' is error, then,
y1 = f(x1) + e1 {e1 may be a random number, may be 0 also}
y2 = f(x2) + e2 {e2 may be a random number, may be 0 also}
y3 = f(x3) + e3 {e3 may be a random number, may be 0 also}
y4 = f(x4) + e4 {e4 may be a random number, may be 0 also}
y5 = f(x5) + e5 {e5 may be a random number, may be 0 also}
y6 = f(x6) + e6 {e6 may be a random number, may be 0 also}
y7 = f(x7) + e7 {e7 may be a random number, may be 0 also}
y8 = f(x8) + e8 {e8 may be a random number, may be 0 also}
y9 = f(x9) + e9 {e9 may be a random number, may be 0 also}
y10 = f(x10) + e10 {e10 may be a random number, may be 0 also}

Y = f(X) + E [based on our initial assumpotions, E is 0]
where, E = e1+e2+e3+e4+e5+e6+e7+e8+e9+10

This can be true only when E is random and Normally distributed and standardized. Where we have a non zero value for individual errors but mean of E will give 0. That's is the reason we are assuming errors are random and normally distributed.

I think the other answers have done a pretty good job of explaining what the assumption means, but I just want to add why this is such an important assumption for any future readers. This implies that outside of the covariates we include in our model, the rest of the variance is completely independent and normally distributed across the observations. However there are actually a lot of times in the real world that this isn't true. Let's say you want to find the relationship between a person's income and their BMI from a dataset of thousands of people from all 50 states in US. Intuitively, we may suspect that people from the same area of the country are probably more similar to each other in both weight and income than people from random parts of the country. Therefore, we can't assume the error is a zero-mean normally independently distributed term.

However, we also probably don't want to add 50 state coefficients or hundreds of county coefficients into our model. Instead we can use random effects hierarchical model where we assume that the states or counties also have their own error distribution and that component of the error term is separated from the individual error term. By doing this we reduce the risk of identifying a spurious relationship between BMI and income which really is due to regional differences in income and BMI. Similar approaches are done when modeling spatial phenomena (i.e. we expect neighboring ZIP codes or counties to be more similar to each other than farther apart ones) and longitudinal analysis (observations within one patient are going to be related to each other).

  • It may not be a direct answer to the question, but it's better than that...It puts the question in context. Big picture is not taught enough in courses. MathSE can help with that! – C Monsour Apr 30 at 2:24

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.