Derivative of pseudoinverse with respect to original matrix I have been trying to find an analytical expression for the following:
$\frac{\partial {X^{+}}}{\partial {X}}$
In my case, $X$ has a constant rank.
I've found the formula for differentiating a pseudoinverse in Goulob's paper (equation 4.12): 
$$
\frac{\mathrm d}{\mathrm d x} A^+(x) =
 -A^+ \left( \frac{\mathrm d}{\mathrm d x} A \right) A^+
+A^+ A{^+}^T  \left( \frac{\mathrm d}{\mathrm d x} A^T \right) (1-A A^+)
+ (1-A^+ A) \left( \frac{\mathrm d}{\mathrm d x} A^T \right) A{^+}^T A^+
$$
but I can't see how to input the original matrix. I've tried
$$
\frac{\mathrm d}{\mathrm d A} A^+(A) =
 -A^+ \left( \frac{\mathrm d}{\mathrm d A} A \right) A^+
+A^+ A{^+}^T  \left( \frac{\mathrm d}{\mathrm d A} A^T \right) (1-A A^+)
+ (1-A^+ A) \left( \frac{\mathrm d}{\mathrm d A} A^T \right) A{^+}^T A^+
$$
but it doesn't seem to give the right answers. What should $\frac{\mathrm d A^T}{\mathrm d A}$ and $\frac{\mathrm d A}{\mathrm d A}$ evaluate to?
 A: A matrix-matrix gradient will be a 4th order tensor. So it's easier to start with the differential, since that is an ordinary matrix. 
For ease of typing, I'll use $P$ for the pseudoinverse of $A$. It's differential is
$$\eqalign{
  dP &= PP^T\,dA^T\,(I-AP) + (I-PA)\,dA^T\,P^TP -P\,dA\,P \cr\cr
}$$
Next we need to factor out $\,dA,\,$ for which we'll need two special 4th order tensors 
$$\eqalign{
{\mathcal B}_{ijkl} = \delta_{il}\,\delta_{jk} \cr
{\mathcal E}_{ijkl} = \delta_{ik}\,\delta_{jl} \cr
}$$
These tensors can be used to transpose matrices and to rearrange the order of matrix products 
$$\eqalign{
X^T &= {\mathcal B}:X \cr
 A\,X\,B &= A{\mathcal E}B^T:X \cr
}$$
where colon represents the inner (Frobenius) product. 
Continuing with the pseudoinverse
$$\eqalign{
  dP
 &= \Big(PP^T{\mathcal E}(I-AP)^T:{\mathcal B} + (I-PA){\mathcal E}P^TP:{\mathcal B} -P{\mathcal E}P^T\Big):dA \cr
\cr
\frac{\partial P}{\partial A}
 &= \Big(PP^T{\mathcal E}(I-AP)^T + (I-PA){\mathcal E}P^TP\Big):{\mathcal B} -P{\mathcal E}P^T \cr\cr
}$$
Interestingly, the answer to your last question is
$$\eqalign{
 {\mathcal B} &= \frac{\partial A^T}{\partial A},\,\,&\,\, {\mathcal E} &= \frac{\partial A}{\partial A} \cr
}$$
