4
$\begingroup$

Building off of my previous question, I am trying to derive the normal equations for the least squares problem:

$$ \min_W \|WX - Y\|_2 \\ W \in \mathbb{C}^{N \times N} \quad X, Y \in \mathbb{C}^{N \times M} $$

The intuitive way of viewing this problem is that I am trying to predict a vector $y$ (of length $N$) from a corresponding $x$ vector using a matrix $W$, and to estimate $W$ I have multiple ($M$, to be precise) realizations of $x$ and $y$ packed into matrices $X$ and $Y$.

I am trying to define this in terms of a least-squares problem and derive the normal equations myself, but I'm running into issues in taking the derivative. To spell it out explicitly, I can re-state the above equation as:

$$ \min_W (WX - Y)^H (WX - Y) \\ = \min_W (X^H W^H - Y^H) (WX - Y) \\ = \min_W (X^H W^H W X - Y^H W X - X^H W^H Y + Y^H Y) $$

Now typically I would take the derivative with respect to $W$, set it equal to zero, and solve for $W$. However, my matrix calculus is rusty and everything I know is basically summed up on this webpage, where it explicitly states:

Note that the Hermitian transpose is not used because complex conjugates are not analytic.

Now, because of Michael C. Grant's answer I feel there must be some way of doing this, but I am at a loss as to how. Thank you all in advance!

$\endgroup$
4
$\begingroup$

In the first version of my answer I overlooked the fact that you defined the $L_2$ error in a way that is not suitable for matrices. For vectors it would have been OK, but if your complex error is $E=WX-Y$, then $E^HE$ is not the error that you want to minimize, because $E^HE$ is a matrix (if $E$ were a vector, then $E^HE$ would be a scalar and you could minimize it). The appropriate definition of the (scalar) error measure is given by the Frobenius norm

$$\epsilon=\|E\|_F^2=\|WX - Y\|_F^2=\\ =\text{trace}\left ( E^HE\right)=\text{trace}\left ( (WX-Y)^H(WX-Y)\right)=\\ =\text{trace}\left ( X^HW^HWX - X^HW^HY - Y^HWX +Y^HY\right)$$

Now we can apply the trick of taking the conjugate complex derivative (reference) w.r.t. $W^H$:

$$\frac{\partial\epsilon}{\partial W^H} = WXX^H - YX^H$$

Setting the derivative to zero gives the solution

$$W=YX^H(XX^H)^{-1}$$

assuming $(XX^H)^{-1}$ exists.

EDIT: just an additional simple example to show how the trick with the conjugate derivative works when solving for an unknown complex vector (instead of a matrix). In this case we get a well-known result:

Assume we have an overdetermined system of complex linear equations

$$Ax=b$$

where $A$ is $m\times n$, $m>n$, $x$ is $n\times 1$, and $b$ is $m\times 1$. Minimizing the squared error

$$\epsilon=(Ax-b)^H(Ax-b)=x^HA^HAx - x^HA^Hb-b^HAx+b^Hb$$

by taking the derivative w.r.t. $x^H$ and equating it with zero gives

$$A^HAx-A^Hb=0$$

which leads to the well-known Moore-Penrose pseudoinverse:

$$x=(A^HA)^{-1}A^Hb$$

EDIT 2: Let's have a look at two little examples showing that it is correct to regard a complex variable $z=z_R+iz_I$ as constant when taking the derivative w.r.t. $z^*$. The conjugate derivative is defined as (ref.)

$$\frac{\partial f}{\partial z^*}=\frac{1}{2}\left [\frac{\partial f}{\partial z_R} +i\frac{\partial f}{\partial z_I} \right]$$

For $f=z$ we get

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R+iz_I)}{\partial z_R} +i\frac{\partial (z_R+iz_I)}{\partial z_I} \right]=\frac{1}{2}(1+i^2)=0$$

For $f=zz^*$ we have

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R^2+z_I^2)}{\partial z_R} +i\frac{\partial (z_R^2+z_I^2)}{\partial z_I} \right]=\frac{1}{2}(2z_R+2iz_I)=z$$

$\endgroup$
  • $\begingroup$ This seems like it's kind of cheating though..... :P I see you've gotten the "correct" answer, but modifying an entry in $W$ will certainly modify it in $W^H$ and vice versa, therefore I don't see how we can ignore one or the other as a "constant" $\endgroup$ – staticfloat May 16 '13 at 21:49
  • $\begingroup$ As I said, it's a pure formalism but it gives the correct result. It has to do with the fact that the function to be minimized is real-valued. In this case the complex gradient is given by the derivative w.r.t. the complex conjugate variable matrix. I actually found an online reference here. Check out section 4! $\endgroup$ – Matt L. May 16 '13 at 21:55
  • $\begingroup$ I've added a simple example with a well-known solution to my answer to make the method a bit more believable ... $\endgroup$ – Matt L. May 16 '13 at 22:35
  • $\begingroup$ You've worked hard for this acceptance, thank you for your effort! $\endgroup$ – staticfloat May 17 '13 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.