# Complex ($\mathbb C$) least squares derivation

I know how to derive the least squares in the real domain.

If a tall matrix $A$ and a column vector $b$ are real, then the solution of the least squares problem $Ax = b$ can be derived as:

\begin{align} \{E(x)\}^2 &= ||Ax - b||^2 \\ &= (Ax-b)^T (Ax-b) \\ &= x^T A^T Ax - x^T A^T b - b^T Ax + b^T b \\ &= x^T A^T Ax - 2 x^T A^T b + b^T b \qquad (\because (Ax)^T b = b^T (Ax)) \end{align}

Differentiating both sides with respect to $x$,

\begin{align} \frac{d \{E(x)\}^2}{dx} &= 2A^T Ax - 2 A^T b \end{align}

Setting $\frac{d \{E(x)\}^2}{dx} = 0$ to find when we get the minimum $E(x)$,

$$2A^T Ax - 2 A^T b = 0 \\ A^T Ax = A^T b \\ x = (A^T A)^{-1} A^T b$$

Now, we turn to the complex-valued situation.
Assume $A$ and $b$ are complex,

\begin{align} \{E(x)\}^2 &= ||Ax - b||^2 \\ &= (Ax-b)^H (Ax-b) \\ &= x^H A^H Ax - x^H A^H b - b^H Ax + b^H b \\ \end{align}

Here, I have some problems.
First, $x^H A^H b \neq b^H Ax$ unless $(Ax)^H b$ is real.
Most of all, I don't know how to differentiate the complex matrices above.

How to proceed the derivation?
There are plenty of derivations in the real domain in Google, but I couldn't find detailed explanation of the general complex case.

• Hint: decompose in real and imaginary parts and differentiate on the real vectors $x_r$ and $x_i$ separately: $\|(A_rx_r-A_ix_i-b_r)+i(A_ix_r+A_rx_i-b_i)\|^2$.
– user65203
May 13, 2016 at 7:13
• Oct 15, 2021 at 1:23

Denote the complex conjugate, transpose, and conjugate transpose of the matrix $$A$$ as $$(A^*, A^T, A^H)$$ respectively.
Use the Frobenius (:) Inner Product to write the function and take its differential \eqalign{ f &= (Ax-b)^*:(Ax-b) \cr\cr df &= (Ax-b)^*:A\,dx \cr &= A^T(Ax-b)^*:dx \cr } Since $$df=\Big(\frac{\partial f}{\partial x}:dx\Big),\,$$ the gradient must be \eqalign{ \frac{\partial f}{\partial x} &= A^T(Ax-b)^* \cr } Set the gradient to zero, take the complex conjugate, and solve for $$x$$ \eqalign{ A^T(Ax)^* &= A^Tb^* \cr A^HAx &= A^Hb \cr x &= (A^HA)^{-1}A^Hb \cr &= A^{+}b \cr } Notice that $$x$$ and $$x^*$$ are treated as independent variables for the purpose of differentiation.
• @Kipton every context. Clearly $(1, 1)$ and $(1, -1)$ are linearly independent, and equally so are $1+i$ and $1-i$ when taken as a vector over $\mathbb R^2$ (one dimensional things are always linearly dependent so it's pointless to talk about $\mathbb C$). Note that the latter is the complex conjugate of the former. You can also verify that $\frac{\textrm d}{\textrm dz}\bar z = 0$.
• Thanks. This finally clicked for me when I saw the formal definition of the Wirtinger derivativative, $\partial / \partial z = (1/2) (\partial / \partial x - i \partial / \partial y)$ where $z = x + i y$, which generalizes the usual complex derivative to non-holomorphic functions. With this definition, you automatically get nice properties like the ones you mentioned, as well as the chain rule (effectively treating $z$ and $\bar z$ as independent). Nov 27, 2021 at 16:53