6
$\begingroup$

I know how to derive the least squares in the real domain.

If a tall matrix $A$ and a column vector $b$ are real, then the solution of the least squares problem $Ax = b$ can be derived as:

$$\begin{align} \{E(x)\}^2 &= ||Ax - b||^2 \\ &= (Ax-b)^T (Ax-b) \\ &= x^T A^T Ax - x^T A^T b - b^T Ax + b^T b \\ &= x^T A^T Ax - 2 x^T A^T b + b^T b \qquad (\because (Ax)^T b = b^T (Ax)) \end{align}$$

Differentiating both sides with respect to $x$,

$$\begin{align} \frac{d \{E(x)\}^2}{dx} &= 2A^T Ax - 2 A^T b \end{align}$$

Setting $\frac{d \{E(x)\}^2}{dx} = 0$ to find when we get the minimum $E(x)$,

$$ 2A^T Ax - 2 A^T b = 0 \\ A^T Ax = A^T b \\ x = (A^T A)^{-1} A^T b $$

Now, we turn to the complex-valued situation.
Assume $A$ and $b$ are complex,

$$\begin{align} \{E(x)\}^2 &= ||Ax - b||^2 \\ &= (Ax-b)^H (Ax-b) \\ &= x^H A^H Ax - x^H A^H b - b^H Ax + b^H b \\ \end{align}$$

Here, I have some problems.
First, $x^H A^H b \neq b^H Ax$ unless $(Ax)^H b$ is real.
Most of all, I don't know how to differentiate the complex matrices above.

How to proceed the derivation?
There are plenty of derivations in the real domain in Google, but I couldn't find detailed explanation of the general complex case.

$\endgroup$
2
  • 1
    $\begingroup$ Hint: decompose in real and imaginary parts and differentiate on the real vectors $x_r$ and $x_i$ separately: $\|(A_rx_r-A_ix_i-b_r)+i(A_ix_r+A_rx_i-b_i)\|^2$. $\endgroup$
    – user65203
    May 13, 2016 at 7:13
  • $\begingroup$ related: stats.stackexchange.com/questions/66088/… $\endgroup$ Oct 15, 2021 at 1:23

1 Answer 1

8
$\begingroup$

Denote the complex conjugate, transpose, and conjugate transpose of the matrix $A$ as $(A^*, A^T, A^H)$ respectively.

Use the Frobenius (:) Inner Product to write the function and take its differential $$\eqalign{ f &= (Ax-b)^*:(Ax-b) \cr\cr df &= (Ax-b)^*:A\,dx \cr &= A^T(Ax-b)^*:dx \cr }$$ Since $df=\Big(\frac{\partial f}{\partial x}:dx\Big),\,$ the gradient must be $$\eqalign{ \frac{\partial f}{\partial x} &= A^T(Ax-b)^* \cr }$$ Set the gradient to zero, take the complex conjugate, and solve for $x$ $$\eqalign{ A^T(Ax)^* &= A^Tb^* \cr A^HAx &= A^Hb \cr x &= (A^HA)^{-1}A^Hb \cr &= A^{+}b \cr }$$ Notice that $x$ and $x^*$ are treated as independent variables for the purpose of differentiation.

$\endgroup$
3
  • $\begingroup$ "Notice that 𝑥 and 𝑥∗ are treated as independent variables for the purpose of differentiation." -- It's clear that this works out here, but what is the intuition for treating x and x* as independent? In what contexts does this trick work? $\endgroup$ Oct 26, 2021 at 15:02
  • $\begingroup$ @Kipton every context. Clearly $(1, 1)$ and $(1, -1)$ are linearly independent, and equally so are $1+i$ and $1-i$ when taken as a vector over $\mathbb R^2$ (one dimensional things are always linearly dependent so it's pointless to talk about $\mathbb C$). Note that the latter is the complex conjugate of the former. You can also verify that $\frac{\textrm d}{\textrm dz}\bar z = 0$. $\endgroup$
    – Zxv
    Nov 25, 2021 at 7:18
  • $\begingroup$ Thanks. This finally clicked for me when I saw the formal definition of the Wirtinger derivativative, $\partial / \partial z = (1/2) (\partial / \partial x - i \partial / \partial y)$ where $z = x + i y$, which generalizes the usual complex derivative to non-holomorphic functions. With this definition, you automatically get nice properties like the ones you mentioned, as well as the chain rule (effectively treating $z$ and $\bar z$ as independent). $\endgroup$ Nov 27, 2021 at 16:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .