Expanding squared L2 norm of difference of two vectors and differentiating I'm confused on how to expand and differentiate a squared $L^2$-norm when more than one vector and a matrix is involved. For example:
$$\nabla_x(\| z-Zx\|^2_2)$$
where $z \in \mathbb{R}^n$, $Z \in \mathbb{R}^{n\times n}$  and $x = [x_1,x_2,\ldots,x_n]^T \in \mathbb{R}^n$.
For the case where $\nabla_x(\cdot)$ is just $\mid\mid x\mid \mid^2_2$, it's fairly easy in that you apply the definition of the $L^2$-norm, square, and differentiate which would yield $2x$. What properties can be used here to make this problem solvable?
EDIT: The solution set claims that the expression above equates to:
$2(Z^TZx-Z^Tz)$
My original question still stands
 A: Well, while it is true that under some noation $\nabla_x \|x\|^2_2=2x$, I think it is a bad habit as it doesn't go natuarlly with matrix calculus. Usually, given $f:\mathbb R^n\rightarrow\mathbb R^m$, its derivative matrix $Df$ is an $m\times n$ matrix. The $ij$ entry is the derivative of $f_i$ w.r.t $x_j$. So in this case we expect $D(\|x\|^2_2)$ to be an $1\times n$ row vector. Namely, $D(\|x\|^2_2)=2x^T$.
The chain rule only works naturally in this setting, when we are using derivative matrices. It says that (under some simplification), $D(f\circ g) (x)= Df(g(x))\cdot Dg(x)$ The multiplication is matrix multiplication. Under this setting we see the following: I am using the well known formula $D(Ax)=A$ (which makes sense, a linear approximation of a linear function is itself)
$$D_x(\|z-Zx\|_2^2)=2(z-Zx)^T D(z-Zx)=-2(z^T-x^TZ^T)Z$$
This is a row vector of course. I am guessing your book equate it to zero. In this case, to get a nicer notation you may transpoe both sides and get $Z^T(z-Zx)=0$. Other approaches would be using the fact that the squared norm is just $(Zx-z)^T(Zx-z)$, foiling and then there's no need to use the chain rule.
A: Let's write the squared norm in this form and use $y$ instead of $z$ to avoid confusion
\begin{equation}
\begin{split}
f & = \|y - Zx\|^2_2 \\
& = (y - Zx)^T(y - Zx) \\
& = (y^T - x^TZ^T)(y - Zx) \\
& = y^Ty - y^TZx - x^TZ^Ty + x^TZ^TZx \\
df & = d(y^Ty) - d(y^TZx) - d(x^TZ^Ty) + d(x^TZ^TZx)  
\end{split}
\end{equation}
Now we will work out each term separately,
It is clear that $\frac{d(y^Ty)}{dx} =0$, so no need to develope it further
For the 2nd term $d(y^TZx)$,
\begin{equation}
\begin{split}
d(y^TZx) & =  (dy^T)Zx + y^T(dZ)x + y^TZ(dx) \\
\frac{d(y^TZx)}{dx} & =  y^TZ \\
\end{split}
\end{equation}
For the 3rd term $d(x^TZ^Ty)$, we will use this property
$$ d(y^TZx) = d(y^TZx)^T = d(x^TZ^Ty) $$
so
$$ \frac{d(x^TZ^Ty)}{dx} = y^TZ $$
For the last term $d(x^TZ^TZx)$, we have just to put $x=y$ and differentiate, so
\begin{equation}
\begin{split}
d(x^TZ^TZx) & = (x^TZ^TZ)dx + (x^TZ^TZ)dx \\
\frac{d(x^TZ^TZx)}{dx} &= x^T(Z^TZ + Z^TZ) \\
&= 2x^T(Z^TZ) \\
\end{split}
\end{equation}
Note: Depending on your preferred Layout convention, the derivative could be either
$$\frac{d(x^TZ^TZx)}{dx} = x^T(Z^TZ + (Z^TZ)^T) = 2 x^TZ^TZ$$
or
$$\frac{d(x^TZ^TZx)}{dx} = 2Z^TZx$$
Finally, putting all terms together, we obtain
$$ d(\|y - Zx\|^2_2) = 2x^T(Z^TZ) - 2y^TZ = 2(x^TZ^T -y^T)Z$$
using other convention, we obtain
$$ d(\|y - Zx\|^2_2) = 2Z^TZx - 2Z^Ty = 2Z^T(Zx -y)$$
A: $\def\p{\partial}$
For ease of typing, define the vector
$$w=Zx-z$$
Write the function in terms of this new vector, then calculate its gradient.
$$\eqalign{
 \lambda &= \|w\|^2 = w^Tw \\
d\lambda &= 2w^Tdw = 2w^TZ\,dx = (2Z^Tw)^Tdx \\
\frac{\p\lambda}{\p x} &= 2Z^Tw = 2(Z^TZx - Z^Tz) \\
}$$
Exactly as claimed by the solution set. Yay!
