Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I believe the most restrictive assumption we can place on a series of observations is that they are iid.

It is possible to relax these assumptions. For example relaxing the independent distribution results in independent heterogeneously distributed random variables. In other words the distribution of each random variables can itself vary. What does this mean? There must be some importance or we would not bother to specify the properties of the distribution. Can the distribution depend on preceding observations or would this break the independent observation restriction that we still have in place?

Can anyone think of any intuitive examples to help explain this (in the same way that tossing a fair coin or rolling a fair die are good examples of iid observations) and any important statistical concepts that arise out of relaxing this assumption? My degree is not in statistics although I am quite interested in this subject. Thank you all so much.

share|improve this question
"relaxing the independent distribution results in independent heterogeneously..." Perhaps you mean "relaxing the identical ..." ? –  leonbloy Aug 29 '13 at 14:20

3 Answers 3

Some parts of your question are not clear, but if you want examples in statistics in which weaker assumptions than i.i.d. can bear fruit, here are two:

  • The weak law of large numbers says that if $X_1,X_2,X_3,\ldots$ are uncorrelated (a weaker assumption than independence) random variables each with expected value $\mu$ and variance $\sigma^2$ (a weaker assumption that indentical distribution) then $$ \forall\varepsilon>0\ \lim_{n\to\infty} \Pr\left(\left|\frac{X_1+\cdots+X_n}{n}-\mu\right| < \varepsilon \right) = 1. $$

  • The Guass--Markov theorem considers a model $Y=X\beta+\varepsilon$ where $X\in\mathbb R^{n\times p}$ with $n\gg p$, $\beta\in\mathbb R^{p\times 1}$, $\varepsilon\in\mathbb R^{n\times 1}$ and so $Y\in\mathbb R^{n\times 1}$, and $X$ and $Y$ are observable data, $\beta$ unobservable and fixed, and $\varepsilon$ is unobservable and random. "Fixed" means not random; and "random" means something that changes if a new sample is taken. Thus $\varepsilon$ and hence $Y$ change if a new sample is taken, which the observable $X$ and the unobservable $\beta$ remain the same.

    One assumes the entries in $\varepsilon$ have expected value $0$ and all have the same variance $\sigma^2<\infty$ (a weaker assumption that indentical distribution) and they are uncorrelated (a weaker assumption than independence).

    The conclusion is that the best linear unbiased estimator (B.L.U.E.) of $\beta$ is $\hat\beta=(X^TX)^{-1}X^T Y$, the least-squares estimator. ("Linear" means linear as a function of $Y$.)

share|improve this answer
Can the person who down-voted this explain? –  Michael Hardy May 24 '14 at 23:59

Relaxing the assumptions doesn't necessarily mean that you go from assuming a normal variable in period $t$, and a binomial in period $t+1$, or something similarly drastic. You are typically dealing with a parametric probability distribution; eg $\mathcal{N}(\mu, \sigma)$, which you would change to $\mathcal{N}(\mu_t, \sigma_t)$, or also, say, to something like $\mathcal{N}(\mu, \sigma_{f(x_0, ..., x_{t-1})})$, among many other variations. The last case is essentially what is done in models very commonly used in financial time series modeling, so-called ARCH models and their variants (c., e.g., http://en.wikipedia.org/wiki/Autoregressive_conditional_heteroskedasticity, or http://www.amazon.com/Time-Analysis-James-Douglas-Hamilton/dp/0691042896). Why this is useful depends on the case. When you look at historic time series, and sticking to this last example, you notice that the variance of certain time series tends to change over time. Hence, such a model would fit the data better than an iid assumption.

You have all kind of variations of the basic iid assumption that is originally driven from trying to fit real-world data better. To mention another example, if you have a time series in which crashes or booms occur (be that the stock market or simply a country's relative development), you might look at time-series which inherently will have discontinuous jumps, such as so-called regime-switch models (eg., http://en.wikipedia.org/wiki/SETAR_(model)).

All these models are considered useful, are widely employed, and well-investigated; so it should be easy to follow a google trail. If you were looking for book recommendations, I enjoyed the above-linked book by Hamilton which, when I was in grad school, was the standard (but this is over ten years ago). It is non-measure-theoretic, but rigorous, and was then current with research (I don't know if Hamilton kept updating it after).

share|improve this answer

An example of a random process yielding multiple iid random variables could be one of the dimensions of some widget coming off an automated assembly line. We want this dimension of the widget to be within a certain tolerance of a certain value, but the machines that make it produce widgets with a variety of sizes of this dimension.

Let $X_k$ be the size of the $k$th widget. It's reasonable to assume that the outcome of producing one widget does not affect the dimensions of any other widget (hence the $X_k$ are independent), and that the machines neither get better nor worse at producing widgets of the correct size (hence the $X_k$ are identically distributed).

In linear regression analysis, where you are trying to fit a straight line to a set of observations of some outcome $y$ that you think depends linearly on some input $x$, it is generally assumed that each $y_k$ is some linear function of its input $x_k$ plus a random variable, and that the random variable is the same for every observation.

If you relax this assumption by saying that the distribution of the random variables are not identical but in fact the distribution is somehow a function of the input value $x$, for example the variance is larger when $x$ is larger, then your data have a property called heteroscedasticity. This can be a serious problem, and much has been written about how to deal with it. It is more difficult to do a good analysis of the data in this case than when the data for the linear regression do not have this property.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.