I have vectors $x \in \mathbb{R}^n$ and I expect some multivariate normal distribution.

I want to normalize the vectors in such a way that $y = M (x - b)$ has mean zero ($\operatorname{E}[Y] = 0$) and variance 1 ($\operatorname{var}(Y) = 1_n$).

For $n=1$, that is basically the Standard score.

  1. With $M = 1_n$ and $b = \operatorname{E}[X]$, I get it mean-normalized but not variance-normalized.

  2. With $M$ being a non-negative symmetric square root of $\operatorname{var}(X)^{-1}$, we get $\operatorname{var}(Y) = 1_n$. We can choose $b$ as in case 1.

I want to estimate $M$ and $b$ based on some running/moving algorithm. For the first case, this is easy:

For every new $x$, I can use the update rule

$$b \leftarrow x \alpha + b (1 - \alpha)$$

for some $\alpha$. I could set $\alpha = 1/N$ and set $N \leftarrow N + 1$ in every iteration, starting with $N = 1$.

How would I do something similar for the second case, esp. for $M$?

I could use a similar update rule to calculate the covariance matrix, e.g.

$$\Sigma \leftarrow (x - b) (x - b)^T \alpha + \Sigma (1- \alpha) $$

but I'm not sure how well that works in practice (also, I haven't really seen that formula in such form) or if that even converges (given that it uses also the moving $b$).

Also, if I use this, I only have an estimation of $\operatorname{var}(X)$ but I still need the $M$. I probably could use another estimation to get the $M$ based on the $\Sigma$ (which one?), but I wonder if I could just get an estimation of $M$ more directly.

(I asked the same question also on MetaOptimize.)


I assume you want to avoid matrix inversion at each step? There are two possibilities:

I. Estimate inverse covariance directly

If you average the inverse of X*X^T instead of taking the inverse of the average of X*XT, then y will not be a normal random variable. Still, it can asymptotically look like one. See e.g.:

P. Shaman, "An approximate inverse for the covariance matrix of moving average and autoregressive processes" http://projecteuclid.org/download/pdf_1/euclid.aos/1176343085

II. Estimate covariance and invert with iterative method

Estimate covariance in the usual way:

Using homogeneous coordinates Z = [x0, x1, .., xn, 1] Keep a running weighted sum of S = Z * Z^T The mean Bj = Snj / Snn for j < n The covariance Cij = Sij / Snn for i, j < n

Then since Cij changes slowly over time, compute at each step the inverse covariance M = Inv(C) given a fast iterative method by remembering the prior Mprev, e.g.

F. Soleymani, "On a fast iterative method for approximate inverse of matrices" http://www.mathnet.or.kr/mathnet/thesis_file/CKMS-28-2-407-418.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.