Suppose data set is expressed by the matrix $X \in\mathbb R^{n \times d}$

where $n =$ Number of samples and $d =$ dimension/features of each sample

Then what does $\operatorname{Cov}(X) \in\mathbb R^{d \times d}$ (Variance-Covariance matrix of $X$) represent. Does below interpretation would be right

Variance-Covariance matrix of $X$ represents covariance between every pair of dimension/feature for all samples.


Let's review how the covariance matrix is computed in this context. Let $\mu_j$ denote the mean of the $j$th column, and let $\mu$ denote the row-vector $\mu = (\mu_1,\mu_2,\dots,\mu_d)$. Then $$ \operatorname{cov}(X) = (X - \mu 1_n)(X - \mu 1_n)^T $$ where $1_n$ denotes the column vector $(1,\dots,1)^T$ of length $n$.

With this in mind, the $i,j$ entry of the covariance matrix is given by $$ \operatorname{cov}(X)[i,j] = \sum_{k=1}^n (x_{ik} - \mu_k)(x_{jk} - \mu_k) $$ So, $\frac 1n \operatorname{cov}(X)[i,j]$ is the covariance between the $i$th feature and $j$th feature, and $\frac 1n \operatorname{cov}(X)[i,i]$ is the variance of the $i$th feature.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.