Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.
Sign up to join this community
Suppose data set is expressed by the matrix $X \in\mathbb R^{n \times d}$
where $n =$ Number of samples and $d =$ dimension/features of each sample
Then what does $\operatorname{Cov}(X) \in\mathbb R^{d \times d}$ (Variance-Covariance matrix of $X$) represent. Does below interpretation would be right
Variance-Covariance matrix of $X$ represents covariance between every pair of dimension/feature for all samples.
Let's review how the covariance matrix is computed in this context. Let $\mu_j$ denote the mean of the $j$th column, and let $\mu$ denote the row-vector $\mu = (\mu_1,\mu_2,\dots,\mu_d)$. Then $$ \operatorname{cov}(X) = (X - \mu 1_n)(X - \mu 1_n)^T $$ where $1_n$ denotes the column vector $(1,\dots,1)^T$ of length $n$.
With this in mind, the $i,j$ entry of the covariance matrix is given by $$ \operatorname{cov}(X)[i,j] = \sum_{k=1}^n (x_{ik} - \mu_k)(x_{jk} - \mu_k) $$ So, $\frac 1n \operatorname{cov}(X)[i,j]$ is the covariance between the $i$th feature and $j$th feature, and $\frac 1n \operatorname{cov}(X)[i,i]$ is the variance of the $i$th feature.
