# Get covariance of pixels in an image?

What is the best way to calculate the covariance matrix for a small square of pixels? Assume it is a 3x3 square of grayscale values.

I have read that in some cases, you can get the covariance matrix by inverting the Hessian. So, I was thinking of using a gradient operator to estimate the second derivatives ($J_{xx}, J_{yy}, J_{xy}$). Then, I believe the covariance would be this: $$\frac1{J_{xx}J_{yy} - J_{xy}^2} \begin{bmatrix}J_{yy} & -J_{xy} \\-J_{xy} & J_{xx}\end{bmatrix}$$ Again, I'm not sure if that's correct. Alternatively, I was considering just trying to apply the covariance formula directly. I believe to do that, I'd have each pixel's center $(x,y)$ coordinate be a separate sample. Then calculate covariance, weighting each sample by the pixel color.

What I ended up doing is assigning each pixel becomes a vector $x$ based on its pixel coordinates: $$\begin{matrix} (-1,-1)&(0,-1)&(1,-1)\\ (-1,0)&(0,0)&(1,0)\\ (-1,1)&(0,1)&(1,1) \end{matrix}$$ Then, using the color of each pixel as the weight $w$, you can compute the matrix with (see wikipedia): $$n = \sum_{i=0}^8 w_i\\ m = \sum_{i=0}^8 w_i^2\\ \overline x = \frac1n\sum_{i=0}^8 w_ix_i\\ Cov = \frac n{n^2-m}\sum_{i=0}^8 w_i(x_i-\overline x)^T(x_i-\overline x)$$