Gradient of L2 norm squared for complex vectors Here's what I usually do to check if my gradient is correct:
Let a scalar valued function $f(\textbf{x})$ have a vector valued gradient $\textbf{g(x)}$. Let $\textbf{h}$ be a small perturbation to $\textbf{x}$. Writing the Taylor series for $f(\textbf{x}+\textbf{h})$:
$f(\textbf{x}+\textbf{h}) = f(\textbf{x}) + \textbf{g}(\textbf{x})\textbf{h} + O(\textbf{h}^2)$
I define a quantity $p(\textbf{x},\textbf{h})$ as:
$p(\textbf{x},\textbf{h}) = f(\textbf{x}+\textbf{h}) - f(\textbf{x}) - \textbf{g}(\textbf{x})\textbf{h}$
$r_{h_1} = \frac{p(\textbf{x},\textbf{h})}{p(\textbf{x},\textbf{h})}, r_{h_2} = \frac{p(\textbf{x},2\textbf{h})}{p(\textbf{x},\textbf{h})} \ldots , r_{h_n} = \frac{p(\textbf{x},n\textbf{h})}{p(\textbf{x},\textbf{h})}$
If I've got the gradient right and the second order terms are much larger than the third and higher order terms, then $r_{h_n}$ goes as $n^2$ which can be easily verified.
I used this method to check if the gradient of the function $f(\textbf{x}) = ||\textbf{x}||^2$ given by $\textbf{g}(\textbf{x})=2\textbf{x}$ is correct for a real valued vector $\textbf{x}$ and it was. But when I tried to use the same method for a complex vector, it fails.
I don't see why the gradient for a complex vector would be different. Is it different? If not, what do you think is wrong with my gradient-checking technique?
Edit: My gradient-check failed because $||\textbf{x}|| \neq \Sigma_k\textbf{x}_k^2$ for complex $\textbf{x}_k$. The gradient isn't as simple as $2\textbf{x}$, but I still haven't figured out what the gradient expression is.
Edit2: Deriving from the basic definitions, one can see that the gradient is $x + x^*$. Passing this to my gradient-check function worked. But I am not 100% sure if this is correct.
 A: $
\def\p{\partial}
\def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\grad#1#2{\frac{\p #1}{\p #2}}
$The squared norm of a complex vector is
$$f= \|x\|^2_2 \;=\; x\cdot x^*$$
This is not a function of a single variable $(x)$, but of two variables $(x$ and $x^*)$.
Therefore you must differentiate with respect to each
variable independently, i.e.
$$\eqalign{
df &= x^*\cdot dx + x\cdot dx^* \\
}$$
and this is the relationship that you'll want verify numerically.
But this means that the norm has two different gradients
$$\eqalign{
\grad{f}{x}=x^* \qquad{\rm and}\qquad  \grad{f}{x^*}=x \\
}$$
This approach is referred to as either
Wirtinger derivatives or the ${\mathbb{CR}}$-Calculus
A: Let $\mathrm{U}, \mathrm{V}$ be two normed vector spaces over $\mathbf{K},$ where $\mathbf{K}$ is either $\mathbf{R}$ or $\mathbf{C}.$ We say that a function $f:\mathrm{U} \to \mathrm{V}$ is differentiable at a point $u \in \mathrm{U}$ if the following expansion holds for $h$ in a neighbourhood of zero
$$
f(u + h) = f(u) + Lh + o(h),
$$
where $L$ is a $\mathbf{K}$-linear continuous function. It can be shown that $L$ is uniquely determined by $f$ and the topologies induced by the norms of $\mathrm{U}$ and $\mathrm{V}.$ Given this fact, we write $L = f'(u):\mathrm{U} \to \mathrm{V},$ which is a continuous linear function over the field $\mathbf{K}.$
Suppose now that $\mathbf{K} = \mathbf{C}.$ Since $\mathbf{R} \subset \mathbf{C},$ we see at once that $\mathrm{U}$ and $\mathrm{V}$ can be considered $\mathbf{R}$-vector spaces and that every $\mathbf{C}$-linear function is also a $\mathbf{R}$-linear function. However, when the opposite happens, says $\mathrm{U}$ and $\mathrm{V}$ are real vector spaces then (1) it could happen that they cannot be endowed a complex vector space structure, but even if it did (2) the set of $\mathbf{C}$-linear function is much smaller than the set of $\mathbf{R}$-linear functions. As an example, take $\mathbf{C} = \mathbf{R}^2$ and the function $z \mapsto \bar{z},$ this is not $\mathbf{C}$-linear but it is $\mathbf{R}$-linear since it is the function $(x,y) \mapsto (x, -y).$ In fact, as you may know, every complex number takes the form $a = \rho e^{i \theta}$ and so multiplication by a complex scalar means dilation and rotation (a comformal mapping). It is no surprise that, despite the definition of derivative being the same, that the complex derivative behaves so differently from tis real counterpart since it is the case that you are considering real dilations and rotations, which are very rigid.
I hope this clarifies some doubts.
