Minimizing of expression with norm, vector and matrix I have to derive an analytic solution to:
$$ \min_\lambda||\nabla f(x_k)-A^T \lambda ||^2 $$
with $\lambda \in R^m$, $A \in R^{m \times n}$ and $x_k \in R^n$
I think I then have to derive this expression with respect to $\lambda$ and after that set it to $0$ and isolate $\lambda$, so I think I get a vector for $\lambda$ as solution. But I'm not sure how to deal with the norm and matrix together. Can anyone help me with it?
 A: Let $u=\nabla f(x_k)\in\mathbb{R}^n$, we get
$$
\begin{align*}
\| u - A^T\lambda\|^2 &= (u-A^T\lambda)^T(u-A^T\lambda), \\
&= (u^T-\lambda^TA)(u-A^T\lambda),\\
&= u^Tu-u^TA^T\lambda- \lambda^TAu+\lambda^TAA^T\lambda ,\\
&= \lambda^TAA^T\lambda - 2u^TA^T\lambda + u^Tu.
\end{align*}
$$
Now, let $g(\lambda_1,\lambda_2,\cdots,\lambda_n) = \lambda^TAA^T\lambda - 2u^TA^T\lambda + u^Tu$, we need to take the derivative of $g(\lambda)$, we get
$$
\frac{\partial g}{\partial\lambda_i} = \frac{\partial \lambda^T}{\partial \lambda_i} AA^T\lambda + \lambda^TAA^T  \frac{\partial \lambda}{\partial \lambda_i} -2 u^T A^T \frac{\partial \lambda}{\partial \lambda_i}. 
$$
Since $\frac{\partial \lambda}{\partial \lambda_i}=e_i$ where $e_i=1$, we get
$$
\begin{align*}
\frac{\partial g}{\partial\lambda_i} &= e_i^T AA^T\lambda + \lambda^TAA^T  e_i -2 u^T A^T e_i, \\
&= 2e_i^T AA^T\lambda- 2 e_i^T A u.
\end{align*}
$$
Setting $\frac{\partial g}{\partial \lambda_i}=0$, we get
$$
AA^T\lambda= Au \implies \lambda = (AA^T)^{-1} Au, 
$$
where $AA^T$ is invertible if and only if $A$ is full row rank.
