# Understanding Variance-Covariance Matrix

Suppose data set is expressed by the matrix $$X \in\mathbb R^{n \times d}$$

where $$n =$$ Number of samples and $$d =$$ dimension/features of each sample

Then what does $$\operatorname{Cov}(X) \in\mathbb R^{d \times d}$$ (Variance-Covariance matrix of $$X$$) represent. Does below interpretation would be right

Variance-Covariance matrix of $$X$$ represents covariance between every pair of dimension/feature for all samples.

Let's review how the covariance matrix is computed in this context. Let $$\mu_j$$ denote the mean of the $$j$$th column, and let $$\mu$$ denote the row-vector $$\mu = (\mu_1,\mu_2,\dots,\mu_d)$$. Then $$\operatorname{cov}(X) = (X - \mu 1_n)(X - \mu 1_n)^T$$ where $$1_n$$ denotes the column vector $$(1,\dots,1)^T$$ of length $$n$$.
With this in mind, the $$i,j$$ entry of the covariance matrix is given by $$\operatorname{cov}(X)[i,j] = \sum_{k=1}^n (x_{ik} - \mu_k)(x_{jk} - \mu_k)$$ So, $$\frac 1n \operatorname{cov}(X)[i,j]$$ is the covariance between the $$i$$th feature and $$j$$th feature, and $$\frac 1n \operatorname{cov}(X)[i,i]$$ is the variance of the $$i$$th feature.