In many sources on data analysis, the author(s) talk about calculating covariance of the data, and the formula is given as such

$$ \Sigma = cov(X) = E[(X-E[X])(X-E[X])^T]$$

This formulation is given without any reference to any distribution. And as example, the author can go right to a programming environment, say R, and for famous data set Iris, and do

enter image description here

I am curious if the disconnect between specific distribution and data is an omission, or if covariance calculation on data, any data, can be performed for any distribution. As it happens, multivariate normal is

$$ f(x;\mu,\Sigma) = \frac{ 1}{(2\pi)^{k/2} \det(\Sigma)^{1/2}} \exp \bigg\{ -\frac{ 1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu) \bigg\} $$

and I wonder if $\Sigma$ of the distribution is related to covariance calculated from data. For example maximum likelihood estimator for $\Sigma$ is

$$ \frac{1}{n} \sum_{j=1}^{n} (x_j-\bar{x})(x_j-\bar{x})^T $$

This statement is almost similar to the one using $E[\cdot]$, except we have summing and dividing by $n$ which can be seen as the calculation of expectation?

I have also seen expectation and variance (covariance in multidimensional case) being talked about / derived utilizing method of moments, which makes no assumption on the underlying distribution. I guess I am trying to understand if all of the "summary statistics" have any theoretical basis (i.e. tied to a probability distribution)

Any clarification or pointers are welcome.


The short answer is no, the covariance is a general property, just like the expected value, variance etc. The formula for the expected value is not dervied from any distribution, but is more of a definition, like the moments and central moments.

I don't know if a the sample covariance matrix is an unbiased estimator for a given sample size, but I do know that it is a consistent estimator, and hence will approach the true covariance matrix as you collect more data.

  • $\begingroup$ Actually I think sample covariance and maximum likelihood estimator for $\Sigma$ are exactly the same thing.. Isnt expected value also a max likeli. estimator for $\mu$? $\endgroup$ – BBDynSys Feb 6 '14 at 13:26
  • $\begingroup$ @BB_ML - For a multivariate normal, I believe you are correct. This paper may be helpful to you: statistics.stanford.edu/sites/default/files/2009-10.pdf $\endgroup$ – user76844 Feb 6 '14 at 14:43
  • $\begingroup$ OK.. Knowing this connection is useful; IMO whenever the sample covariance calculation is used, the underlying assumption is data is distributed (multivariate) normally; and if this assumption does not hold, the sample covariance calculation should not be used. What do you think? $\endgroup$ – BBDynSys Feb 6 '14 at 15:43
  • 1
    $\begingroup$ @BB_ML I would generally agree as most standard multivariate stats use covaraince matrix assuming normality (at least asymptotically). Also, covariance is a measure of the degree of linear covariation, so again the normal makes sense. However, it IS possible to induce corelations between almost any pair of random variables, regardless of distribution. Look up "copulas" and you'll see that covariance can be induced for non-normal data, although the interpreation of the matrix is more difficult. The multivariate normal is nice becasue the covariacne is "baked into" the density. $\endgroup$ – user76844 Feb 6 '14 at 16:00
  • $\begingroup$ I see; you are sayhing corelation formula shared above is somewhat a natural fit for multivar normal, it is "baked in", but it can be used for any pair of random vars. That's why you mentioned the formula is a definition. OK. $\endgroup$ – BBDynSys Feb 6 '14 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.