Meaning of off-diagonal multivariate covariance matrices

My terminology might be a bit sloppy. I apologize in advance.

I'm reading on multivariate probabilistic distributions, particularly on Gaussian normal distribution (in the context of probabilistic robotics). I've encountered the generalization of the covariance $\mu$ to the covariance matrix $\Sigma$, and I'd like to get an intuitive understanding of this generalization. The diagonal terms of the matrix are the covariances (how much they differ from the mean) of the corresponding variable, but what about the off-diagonal terms?

If $X$ and $Y$ are frequently both above their means or both below their means but it seldom happens that one is above its mean and the other below its mean, then there are many cases in which $(X-\mathbb E X)(Y-\mathbb E Y)$ is positive and few in which it is negative. In that case the covariance $\mathbb E((X-\mathbb E X)(Y-\mathbb E Y))$ is positive.
If it frequently happens that $X$ is above its mean and $Y$ below its mean, or $X$ is below its mean and $Y$ above its mean, but seldom happens that both are above or both below, then then there are many cases in which $(X-\mathbb E X)(Y-\mathbb E Y)$ is negative and few in which it is positive. In that case the covariance $\mathbb E((X-\mathbb E X)(Y-\mathbb E Y))$ is negative.