Matrix Calculus in Least-Square method In the proof of matrix solution of Least Square Method, I see some matrix calculus, which I have no clue. Can anyone explain to me or recommend me a good link to study this sort of matrix calculus?
In Least-Square method, we want to find such a vector $x$ such that $||Ax-b||$ is minimized. 
Assume $r=Ax-b$
$\Rightarrow\|r\|^2=x^TA^TAx-2b^TAx+b^Tb$
$\Rightarrow \nabla_x \|r\|^2=2A^TAx-2A^Tb$
In the end we set the gradient to zero and find the minimized solution. I understand the whole idea, but I just don't know how exactly we did matrix calculus here, or say I don't know how to do the matrix calculus here. For example, can anyone tell me how we got those transpose in $\|r\|^2$(By what rule?) and how we got the gradient?(how do we take the gradient exactly in matrix format)?
I'll really appreciate if you can help me out. Thanks!
 A: Below are the matrix/vector derivative rules, you will need.
$$\dfrac{d(x^TBx)}{d x_i} = \dfrac{d}{dx_i}\left(\sum_{j,k} x_j B_{jk}x_k\right) = \sum_{j} x_j B_{ji} + \sum_{k}B_{ik} x_k = \sum_{k}\left(B^T + B\right)_{ik}x_k$$
Hence, we have
$$\dfrac{d(x^TBx)}{d x} = (B^T+B)x$$
Similarly, we have
$$\dfrac{d(c^Tx)}{d x_i} = \dfrac{d}{d x_i}\left(\sum_k c_k x_k\right) = c_i$$
Hence, we have
$$\dfrac{d(c^Tx)}{dx} = c$$
Now you should be able to get what you want.
A: All we need here is multivariable calculus, not matrix calculus. Let $f(x) = (1/2) \| Ax - b \|^2$. Notice that $f(x) = g(h(x))$, where $h(x) = Ax - b$ and $g(u) = (1/2) \|u \|^2$. It can easily be seen that the derivatives of $g$ and $h$ are
$$
h'(x) = A, \qquad g'(u) = u^T.
$$
From the multivariable chain rule, we have 
\begin{align*}
f'(x) &= g'(h(x)) h'(x) \\
&= (Ax - b)^T A.
\end{align*}
If we use the convention that $\nabla f(x)$ is a column vector, then
\begin{align}
\nabla f(x) &= f'(x)^T \\
&= A^T (Ax - b).
\end{align}
A: Well the first step is the definition of $||r||^2$. This is easy
\begin{align}
||r||^2 & = \langle r,r \rangle  = r^T r
\\ &= (Ax-b)^T(Ax-b) = (x^TA^T-b^T)(Ax-b)
\\ &= x^TA^TAx -x^TA^Tb-b^TAx +b^Tb
\\  &= x^TA^TAx -(b^TAx)^T -b^TAx +b^Tb
\end{align}
Since $(b^TAx)$ is a scalar it holds  $(b^TAx)⁼ (b^TAx)^T$
Thus
\begin{align}
||r||^2 & = x^TA^TAx -2b^TAx +b^Tb
\end{align}
And for the derivatives, you could take a look here. Another approach would be to write out the matrix-vector expressions in sumation form and calculate the derivative, then no matrices are involved.
A: The problem is to calculate $\nabla_xL$ given the following 
$$\eqalign{
r &= A\cdot x-b \cr
L &= r\cdot r \cr
}$$
First, calculate the differential. Then change the independent variable from $r\to x$
$$\eqalign{
dL &= 2r\cdot dr \cr
 &= 2r\cdot (A\cdot dx) \cr
 &= (2A^T\cdot r)\cdot dx \cr
\nabla_xL &= (2A^T\cdot r) \cr&= 2A^T\cdot(A\cdot x-b) \cr
}$$
