# Error propagation in a simple linear model (asked by a non math-major researcher).

I have the following linear problem in a matrix form $$Ax=Y$$, where $$A$$ is a coefficient matrix, $$x$$ is a vector of unknown parameters and $$Y$$ is a vector of observed data. Matrix $$A$$ is usually over-determined and often rank deficient; it has 1000s of rows and columns. I find a solution by applying the pseudo-inverse (SVD): $$x=A^{-1}Y$$. I compute a conditioning number (equal about 5) to learn how the input errors propagate to the output errors. Can I determine by looking at matrix $$A$$ if each unknown parameter in vector $$x$$ is resolved well and if there is an unwanted correlation between some $$x_{i}$$ and $$x_{j}$$. I have done tests using some synthetic data and they are good, but here I would like to know if anything can be determined from the matrix $$A$$ alone.

• What exactly is "unwanted correlation" between $x_i$ and $x_j$? Commented Feb 2, 2021 at 15:55
• for example in the sollution, one of the unknowns $x_{i}$ correlates with another $x_{j}$, while they should not correlate - it should be caused by the rank-deficiency in $A$
– SVS
Commented Feb 2, 2021 at 16:00

The covariance has the following transformation property $$$$Cov[A\vec{x}] = A Cov[\vec{x}]A^{\intercal},$$$$ where $$Cov[.]$$ is a covariance matrix. This property can be proved as follows $$$$Cov[A\vec{x}] = E[(A\vec{x} - E[A\vec{x}])(A\vec{x} - E[A\vec{x}])] \\ = E[A(\vec{x} - E[\vec{x}])(\vec{x} - E[\vec{x}])A^{\intercal}] \\ = A E[(\vec{x} - E[\vec{x}])(\vec{x} - E[\vec{x}])]A^{\intercal} \\ = A Cov[\vec{x}]A^{\intercal},$$$$ where $$E[.]$$ is the expectation. I.e. in your case $$$$A Cov[\vec{x}]A^{\intercal} = Cov[\vec{y}],$$$$ so the covariance matrix for $$\vec{x}$$ has the following form $$$$Cov[\vec{x}] = A^{+}Cov[\vec{y}](A^{\intercal})^{+},$$$$ where $$A^{+}$$ is pseudoinverse of $$A$$ Sometimes even better to write the answer in terms of inverse covariance matrices: $$$$Cov^{-1}[\vec{x}] = A^{\intercal}Cov^{-1}[\vec{y}]A.$$$$
There is also well-known inequality that links relative perturbations of the solution $$\vec{x}$$ and relative perturbations of the right hand side $$\vec{y}$$: $$$$\frac{\|\Delta\vec{x}\|}{\|\vec{x}\|} \leq \frac{cond(A)}{1 - cond(A)\frac{\|\Delta A\|}{\|A\|}}\left(\frac{\|\Delta\vec{y}\|}{\|\vec{y}\|} + \frac{\|\Delta A\|}{\|A\|}\right),$$$$ where $$\Delta\vec{y}$$ is the perturbation of the vector $$\vec{y}$$, $$\Delta\vec{x}$$ is the perturbation of $$\vec{x}$$ and $$\Delta A$$ is the perturbation of the matrix $$A$$. If the matrix perturbation is zero, the inequality has a simpler form $$$$\frac{\|\Delta\vec{x}\|}{\|\vec{x}\|} \leq cond(A)\frac{\|\Delta\vec{y}\|}{\|\vec{y}\|}.$$$$
The last two inequalities can be simply obtained using the definition of the condition number $$cond(A)=\|A^{+}\|\|A\|$$.