# Using chain rule to calculate Fréchet derivative of $F(X) = \det(A^T (I - X) A)$

Let $$\mathbb{M}^n$$ be the set of real $$n \times n$$ matrices, and let $$A$$ be a fixed real $$n \times n$$ matrix. Define the function $$F: \mathbb{M}^n \rightarrow \mathbb{R}$$ by $$F(X) = \det( A^T (I-X) A)$$ What is the Fréchet derivative (total derivative) of $$F$$ at a matrix $$X$$?

I am having difficulty working it out explicitly. I know that we can apply a chain rule, as $$F = G \circ H$$ where $$H(X) = A^T(I-X)A$$ and $$G(Y) = \det(Y)$$. Yet, I am confused how to apply the chain rule here. I am looking for an expression for the Fréchet derivative and a derivation would be wonderful as well.

Let $$Y = (I-X),\quad\alpha=\det A$$ Then \eqalign{ F &= \alpha^2\det Y \cr dF &= \alpha^2(\det Y)\,(Y^{-1})^T:dY \,\,= F\,Y^{-T}:(-dX) \cr \frac{\partial F}{\partial X} &= -FY^{-T} = \big(X^T-I\big)^{-1}\,\det\big(A^T(I-X)A\big) \cr } where colon denotes the trace inner product, i.e. $$\,A:B={\rm Tr}(A^TB)$$
If $$A$$ is rectangular, then \eqalign{ Y &= A^T(I-X)A \cr F &= \det Y \cr dF &= (\det Y)\,Y^{-T}:dY \,\,= -F\,Y^{-T}:A^TdX\,A \cr \frac{\partial F}{\partial X} &= -(AY^{-T}A^T)\,\,F = \Big(A\big(A^T(X^T-I)A\big)^{-1}A^T\Big)\,\,\det\big(A^T(I-X)A\big) \cr }
• Thanks for the answer. You used that $A$ was square so $\det (A^T(I-X)A) = \det (A)^2 \det (I-X)$. I'm also wondering about the case when $A$ is rectangular. Do you mind also showing how to apply the chain rule in this case? – Chris Harshaw Apr 24 at 18:29