0
$\begingroup$

I need to find the differential and derivative of $ f: X\rightarrow (I-X(X'X)^{-1}X')$

Now by the product rule I found that the differential of $d(I-X(X'X)^{-1}X')=-(dX(X'X)^{-1}X' + (X)d(X'X)^{-1}X'+X(X'X)^{-1}d(X')$

Now I have issues with finding $d(X'X)^{-1}$ because I am not sure how to use the chain rule here, and with the first term: $(dX(X'X)^{-1}X'$ because it is not in the standard form $A(dX)B$ where $A$ and $B$ are any matrices.

Could anyone show me how to proceed as to find the derivative of this function?

$\endgroup$
1
$\begingroup$

For convenience, define $M=(X^TX)^{-1}X^T\,\,$ then the function is $$\eqalign{ F &= I-XM = \,F^T\cr }$$ And the differentials are $$\eqalign{ dM &= (X^TX)^{-1}dX^T - (X^TX)^{-1}(dX^TX)(X^TX)^{-1}X^T - (X^TX)^{-1}(X^TdX)(X^TX)^{-1}X^T \cr &= (X^TX)^{-1}dX^T - (X^TX)^{-1}\,dX^T\,XM - M\,dX\,M \cr &= (X^TX)^{-1}dX^T\,F - M\,dX\,M \cr \cr dF &= -X\,dM - dX\,M \cr &= -X(X^TX)^{-1}\,dX^T\,F+XM\,dX\,M \,-\, dX\,M\cr &= -M^T\,dX^T\,F^T - F\,dX\,M \cr }$$ Apply ${\rm vec}()$ to both sides, but remember that $\,{\rm vec}(dX^T) \neq dx^T$ even though $\,{\rm vec}(dX) = dx$.

Instead, you must use the Kronecker permuation matrix $P$, which depends on the matrix dimensions, to obtain $\,{\rm vec}(dX^T) = P\,dx$. $$\eqalign{ df &= -(F\otimes M^T)\,P\,dx - (M^T\otimes F)\,dx \cr \frac {\partial\,{\rm vec}(F)}{\partial\,{\rm vec}(X)^T} =\frac {\partial f}{\partial x^T} &= -(F\otimes M^T)\,P - (M^T\otimes F) \cr }$$ This kind of vec/vec result is typical for matrix-by-matrix derivatives.

$\endgroup$
0
$\begingroup$

It suffices to derive a product of $3$ terms. Recall that a derivative is a linear function. Let $X\in M_{n,m}$. Then

$Df_X:H\in M_{n,m}\rightarrow -H(X^TX)^{-1}X^T+X(X^TX)^{-1}(H^TX+X^TH)(X^TX)^{-1}X^T-X(X^TX)^{-1}H^T$

We cannot write this expression in tensor form except if we use the function $H\rightarrow H^T$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.