# Derivative of conjugate transpose of matrix

Building off of my previous question, I am trying to derive the normal equations for the least squares problem:

$$\min_W \|WX - Y\|_2 \\ W \in \mathbb{C}^{N \times N} \quad X, Y \in \mathbb{C}^{N \times M}$$

The intuitive way of viewing this problem is that I am trying to predict a vector $y$ (of length $N$) from a corresponding $x$ vector using a matrix $W$, and to estimate $W$ I have multiple ($M$, to be precise) realizations of $x$ and $y$ packed into matrices $X$ and $Y$.

I am trying to define this in terms of a least-squares problem and derive the normal equations myself, but I'm running into issues in taking the derivative. To spell it out explicitly, I can re-state the above equation as:

$$\min_W (WX - Y)^H (WX - Y) \\ = \min_W (X^H W^H - Y^H) (WX - Y) \\ = \min_W (X^H W^H W X - Y^H W X - X^H W^H Y + Y^H Y)$$

Now typically I would take the derivative with respect to $W$, set it equal to zero, and solve for $W$. However, my matrix calculus is rusty and everything I know is basically summed up on this webpage, where it explicitly states:

Note that the Hermitian transpose is not used because complex conjugates are not analytic.

Now, because of Michael C. Grant's answer I feel there must be some way of doing this, but I am at a loss as to how. Thank you all in advance!

## 1 Answer

In the first version of my answer I overlooked the fact that you defined the $L_2$ error in a way that is not suitable for matrices. For vectors it would have been OK, but if your complex error is $E=WX-Y$, then $E^HE$ is not the error that you want to minimize, because $E^HE$ is a matrix (if $E$ were a vector, then $E^HE$ would be a scalar and you could minimize it). The appropriate definition of the (scalar) error measure is given by the Frobenius norm

$$\epsilon=\|E\|_F^2=\|WX - Y\|_F^2=\\ =\text{trace}\left ( E^HE\right)=\text{trace}\left ( (WX-Y)^H(WX-Y)\right)=\\ =\text{trace}\left ( X^HW^HWX - X^HW^HY - Y^HWX +Y^HY\right)$$

Now we can apply the trick of taking the conjugate complex derivative (reference) w.r.t. $W^H$:

$$\frac{\partial\epsilon}{\partial W^H} = WXX^H - YX^H$$

Setting the derivative to zero gives the solution

$$W=YX^H(XX^H)^{-1}$$

assuming $(XX^H)^{-1}$ exists.

EDIT: just an additional simple example to show how the trick with the conjugate derivative works when solving for an unknown complex vector (instead of a matrix). In this case we get a well-known result:

Assume we have an overdetermined system of complex linear equations

$$Ax=b$$

where $A$ is $m\times n$, $m>n$, $x$ is $n\times 1$, and $b$ is $m\times 1$. Minimizing the squared error

$$\epsilon=(Ax-b)^H(Ax-b)=x^HA^HAx - x^HA^Hb-b^HAx+b^Hb$$

by taking the derivative w.r.t. $x^H$ and equating it with zero gives

$$A^HAx-A^Hb=0$$

which leads to the well-known Moore-Penrose pseudoinverse:

$$x=(A^HA)^{-1}A^Hb$$

EDIT 2: Let's have a look at two little examples showing that it is correct to regard a complex variable $z=z_R+iz_I$ as constant when taking the derivative w.r.t. $z^*$. The conjugate derivative is defined as (ref.)

$$\frac{\partial f}{\partial z^*}=\frac{1}{2}\left [\frac{\partial f}{\partial z_R} +i\frac{\partial f}{\partial z_I} \right]$$

For $f=z$ we get

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R+iz_I)}{\partial z_R} +i\frac{\partial (z_R+iz_I)}{\partial z_I} \right]=\frac{1}{2}(1+i^2)=0$$

For $f=zz^*$ we have

$$\frac{\partial f}{\partial z^*}= \frac{1}{2}\left [\frac{\partial (z_R^2+z_I^2)}{\partial z_R} +i\frac{\partial (z_R^2+z_I^2)}{\partial z_I} \right]=\frac{1}{2}(2z_R+2iz_I)=z$$

• This seems like it's kind of cheating though..... :P I see you've gotten the "correct" answer, but modifying an entry in $W$ will certainly modify it in $W^H$ and vice versa, therefore I don't see how we can ignore one or the other as a "constant" – staticfloat May 16 '13 at 21:49
• As I said, it's a pure formalism but it gives the correct result. It has to do with the fact that the function to be minimized is real-valued. In this case the complex gradient is given by the derivative w.r.t. the complex conjugate variable matrix. I actually found an online reference here. Check out section 4! – Matt L. May 16 '13 at 21:55
• I've added a simple example with a well-known solution to my answer to make the method a bit more believable ... – Matt L. May 16 '13 at 22:35
• You've worked hard for this acceptance, thank you for your effort! – staticfloat May 17 '13 at 22:02