# Error is the Projection on the Noise - Does the intuition carry over from least squares?

My question concerns a generalization of the following:

## Background

It is a well known that the least squares problem (on some Hilbert space) \begin{align*} \|AX-Y \|_2^2 \to \min_{A}! \end{align*} for identification of the coefficient, $$A$$, in the statistical model \begin{align*} Y=XA+W \end{align*} has error \begin{align*} \hat A - A = (X^*X)^{-1}X^*[XA+W]-A=(X^*X)^{-1}X^*W, \end{align*} a fact which has numerous derivations. One suspects that the geometric interpretation may lead to generality.

What one notices of course is that since $$(X^*X)^{-1}X^*$$ is the projection onto the linear space spanned by the columns of $$X$$, the error formula can be interpreted as:

The identification error $$\hat A- A$$ in the least squares problem is the projection of the noise (onto the column space of the observations).

Here, a natural question to ask is the projection with respect to what? We know of course that in the simple problem stated above, this can be analyzed entirely with help of the projection theorem, since $$\|\cdot\|_2$$ is a Hilbertian-norm and the range of $$X$$, a the range of a bounded linear operator, is a closed subspace.

## Actual question

Now to my actual question: If we replace the norm $$\| \cdot \|_2^2$$ with an arbitrary but strongly convex twice Frechét differentiable on a Hilbert space $$H$$, (the point is that it should be a "nice" function) function(al) $$J(\cdot)$$, is it still possible to analyze the error $$\hat A -A$$ in terms of a "projection"? The linear "dynamics" $$Y=XA+W$$ still hold.

What is nice with the formula in the least squares case is that we essentially have \begin{align*} \hat A -A = \Pi_{\mathcal{R}(X)} W. \end{align*}

Rephrasing my question, does there exist a possibly nonlinear operator $$\Gamma$$ which does not depend directly on $$Y$$ such that $$\hat A - A = \Gamma_X(W)$$? If so, what is the regularity of this operator in terms of $$X$$?

## A remark

An approach which might seem fruitful is to consider the problem

\begin{align*} J(AX-Y) \to \min_A! \end{align*} The solution $$A$$ enjoys considerable regularity essentially due to the implicit function theorem. However, I do not know how to, or if it is even possible to, disentangle this implicit function $$\hat A(X,Y)$$ into an operator $$\Gamma_X$$.

Any help, references, or suggestions are much appreciated.

• Have you tried writing out the first-order optimality conditions for $\hat A$? I would if I wasn't on my phone at the moment. – Rahul Nov 23 '18 at 9:55
• I did, but I think I maybe didn't pose the question clear enough before going to stack.exchange, so I didn't see anything from them immediately. Maybe I'm silly. Here goes: If one defines $H(A) = J(AX-Y)$, I guess they are just $H_{A}(A)=X J'(AX-Y) =0$. One also has $H_{AA}(A)=X^*XJ''(AX-Y)$, $H_{AX}=A^*XJ''(AX-Y)$, $H_{AY}=XJ''(AX-Y)$ – sortofamathematician Nov 23 '18 at 10:09
• @Rahul, slight woopsie. $H_{AX}=J'(AX_Y)+A^*XJ''(AX-Y)$, and I'm sure I got every adjoint wrong... – sortofamathematician Nov 23 '18 at 10:17
• Hang on, if you have $Y=XA+W$ and you care about the error $E:=\hat A-A$, then you can just write your objective as $j(X\hat A-Y) = j(X(A+E)-Y) = j(XE-W)$ which does not depend on $Y$. – Rahul Nov 25 '18 at 5:51
• For arbitrary differentiable $j$ the optimality condition is $X^T\nabla j(XE-W)=0$. With the usual least-squares objective $j=\|\cdot\|_2^2$ we have $\nabla j(A)=2A$, which leads to your projection formula, but I'm not sure what more one can say for general $j$. – Rahul Nov 25 '18 at 5:57