Building off of my previous question, I am trying to derive the normal equations for the least squares problem:

$$ \min_W \|WX - Y\|_2 \\ W \in \mathbb{C}^{N \times N} \quad X, Y \in \mathbb{C}^{N \times M} $$

The intuitive way of viewing this problem is that I am trying to predict a vector $y$ (of length $N$) from a corresponding $x$ vector using a matrix $W$, and to estimate $W$ I have multiple ($M$, to be precise) realizations of $x$ and $y$ packed into matrices $X$ and $Y$.

I am trying to define this in terms of a least-squares problem and derive the normal equations myself, but I'm running into issues in taking the derivative. To spell it out explicitly, I can re-state the above equation as:

$$ \min_W (WX - Y)^H (WX - Y) \\ = \min_W (X^H W^H - Y^H) (WX - Y) \\ = \min_W (X^H W^H W X - Y^H W X - X^H W^H Y + Y^H Y) $$

Now typically I would take the derivative with respect to $W$, set it equal to zero, and solve for $W$. However, my matrix calculus is rusty and everything I know is basically summed up on this webpage, where it explicitly states:

Note that the Hermitian transpose is not used because complex conjugates are not analytic.

Now, because of Michael C. Grant's answer I feel there must be some way of doing this, but I am at a loss as to how. Thank you all in advance!

  • $\begingroup$ Can I ask a supplementary question (of Matt L) on this oldish thread. I'm interested in minimization of sum-of-square error (for the specific case of derivation of Frequency-domain Wiener filter). You seem to indicate that the differentiation in this case would be w.r.t the conjugate of the filter (rather than the filter itself) Is this just for convenience - i.e. you could differentiate w.r.t. either? Am I interpreting this correctly, and if so could you point me to a reference on this. $\endgroup$ Apr 17 at 9:20
  • $\begingroup$ Welcome to MSE. This should be a comment, not an answer. $\endgroup$ Apr 17 at 9:44
  • $\begingroup$ If you have further questions on the post please ask a separate question in a new one. This post "space" is meant to be used to reply to the post above, not to post new questions. $\endgroup$
    – Pedro Tamaroff
    Apr 17 at 11:08

In the first version of my answer I overlooked the fact that you defined the $L_2$ error in a way that is not suitable for matrices. For vectors it would have been OK, but if your complex error is $E=WX-Y$, then $E^HE$ is not the error that you want to minimize, because $E^HE$ is a matrix (if $E$ were a vector, then $E^HE$ would be a scalar and you could minimize it). The appropriate definition of the (scalar) error measure is given by the Frobenius norm

$$\epsilon=\|E\|_F^2=\|WX - Y\|_F^2=\\ =\text{trace}\left ( E^HE\right)=\text{trace}\left ( (WX-Y)^H(WX-Y)\right)=\\ =\text{trace}\left ( X^HW^HWX - X^HW^HY - Y^HWX +Y^HY\right)$$

Now we can apply the trick of taking the conjugate complex derivative (reference) w.r.t. $W^H$:

$$\frac{\partial\epsilon}{\partial W^H} = WXX^H - YX^H$$

Setting the derivative to zero gives the solution


assuming $(XX^H)^{-1}$ exists.

EDIT: just an additional simple example to show how the trick with the conjugate derivative works when solving for an unknown complex vector (instead of a matrix). In this case we get a well-known result:

Assume we have an overdetermined system of complex linear equations


where $A$ is $m\times n$, $m>n$, $x$ is $n\times 1$, and $b$ is $m\times 1$. Minimizing the squared error

$$\epsilon=(Ax-b)^H(Ax-b)=x^HA^HAx - x^HA^Hb-b^HAx+b^Hb$$

by taking the derivative w.r.t. $x^H$ and equating it with zero gives


which leads to the well-known Moore-Penrose pseudoinverse:


EDIT 2: Let's have a look at two little examples showing that it is correct to regard a complex variable $z=z_R+iz_I$ as constant when taking the derivative w.r.t. $z^*$. The conjugate derivative is defined as (ref.)

$$\frac{\partial f}{\partial z^*}=\frac{1}{2}\left [\frac{\partial f}{\partial z_R} +i\frac{\partial f}{\partial z_I} \right]$$

For $f=z$ we get

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R+iz_I)}{\partial z_R} +i\frac{\partial (z_R+iz_I)}{\partial z_I} \right]=\frac{1}{2}(1+i^2)=0$$

For $f=zz^*$ we have

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R^2+z_I^2)}{\partial z_R} +i\frac{\partial (z_R^2+z_I^2)}{\partial z_I} \right]=\frac{1}{2}(2z_R+2iz_I)=z$$

  • $\begingroup$ This seems like it's kind of cheating though..... :P I see you've gotten the "correct" answer, but modifying an entry in $W$ will certainly modify it in $W^H$ and vice versa, therefore I don't see how we can ignore one or the other as a "constant" $\endgroup$ May 16 '13 at 21:49
  • $\begingroup$ As I said, it's a pure formalism but it gives the correct result. It has to do with the fact that the function to be minimized is real-valued. In this case the complex gradient is given by the derivative w.r.t. the complex conjugate variable matrix. I actually found an online reference here. Check out section 4! $\endgroup$
    – Matt L.
    May 16 '13 at 21:55
  • $\begingroup$ I've added a simple example with a well-known solution to my answer to make the method a bit more believable ... $\endgroup$
    – Matt L.
    May 16 '13 at 22:35
  • $\begingroup$ You've worked hard for this acceptance, thank you for your effort! $\endgroup$ May 17 '13 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.