Quadratic cost function solution Why the solution to the following cost function:
$$\frac{1}{2}\|Lm-d\|^2 + \frac{1}{2} \mu \|W_m m\|^2_2$$
the below equation:
$$(L^Td+\mu W^T_m W_m)^{-1} L^Td  $$
 A: You can write this as
$\min \frac{1}{2} \| Am-b \|_{2}^{2}$
where
$A=\left[
\begin{array}{c}
L \\
\sqrt{\mu} W_{m} 
\end{array}
\right]$
and
$b=\left[
\begin{array}{c}
d \\
0 
\end{array}
\right]$.
Applying the normal equations to this linear least squares problem gives (assuming that the inverse exists)
$m=(A^{T}A)^{-1}A^{T}b.$
This simplifies to
$m=\left( \left[ L^{T} \sqrt{\mu}W_{m}^{T} \right] \left[ \begin{array}{c}
L \\
\sqrt{\mu}W_{m}
\end{array}\right]\right)^{-1} \left[ L^{T} \sqrt{\mu} W_{m}^{T} \right] \left[ \begin{array}{c}
d \\
0
\end{array}
\right]$
and finally
$m=\left(L^{T}L+\mu W_{m}^{T}W_{m} \right)^{-1}L^{T}d$.
A: Suppose the problem was discretized. I.e. $L$ and $W$ are matrices, $\vec{m}$ and $\vec{d}$ are vectors. Thus the problem is to find the minimum of the following cost function
\begin{equation}
f(\vec{m}) = \frac{1}{2}(L_{ij}m_j - d_i)(L_{ik}m_k - d_i) + \frac{1}{2}\mu W_{ij}m_jW_{ik}m_k
\end{equation}
by arguments $m_p$. To find the extremum of the function, we have to solve the following equation
\begin{equation}
\frac{\partial f}{\partial m_p} = 0, 
\end{equation}
which brings us to the following system of equations
\begin{equation}
(L_{ip}L_{ik} + \mu W_{ip}W_{ik})m_k = L_{ip}d_i.
\end{equation}
The last equation can be rewritten in the matrix form as follows
\begin{equation}
(L^{\intercal}L+\mu W^{\intercal}W)\vec{m}=L^{\intercal}\vec{d}.
\end{equation}
The solution of this equation is
\begin{equation}
\vec{m} = (L^{\intercal}L + \mu W^{\intercal}W)^{-1}L^{\intercal}\vec{d}.
\end{equation}
