What is the best way to calculate the covariance matrix for a small square of pixels? Assume it is a 3x3 square of grayscale values.

I have read that in some cases, you can get the covariance matrix by inverting the Hessian. So, I was thinking of using a gradient operator to estimate the second derivatives ($J_{xx}, J_{yy}, J_{xy}$). Then, I believe the covariance would be this: $$ \frac1{J_{xx}J_{yy} - J_{xy}^2} \begin{bmatrix}J_{yy} & -J_{xy} \\-J_{xy} & J_{xx}\end{bmatrix} $$ Again, I'm not sure if that's correct. Alternatively, I was considering just trying to apply the covariance formula directly. I believe to do that, I'd have each pixel's center $(x,y)$ coordinate be a separate sample. Then calculate covariance, weighting each sample by the pixel color.


1 Answer 1


From my tests, it appears that inverting the Hessian does not give the correct covariance matrix. I'm not sure under what circumstances the inverted Hessian will give the correct answer.

What I ended up doing is assigning each pixel becomes a vector $x$ based on its pixel coordinates: $$ \begin{matrix} (-1,-1)&(0,-1)&(1,-1)\\ (-1,0)&(0,0)&(1,0)\\ (-1,1)&(0,1)&(1,1) \end{matrix} $$ Then, using the color of each pixel as the weight $w$, you can compute the matrix with (see wikipedia): $$ n = \sum_{i=0}^8 w_i\\ m = \sum_{i=0}^8 w_i^2\\ \overline x = \frac1n\sum_{i=0}^8 w_ix_i\\ Cov = \frac n{n^2-m}\sum_{i=0}^8 w_i(x_i-\overline x)^T(x_i-\overline x) $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.