In my current project, I have a function $g(\mathbf{x})$ which I scale with a multivariate exponential term. Now I require a specific partial derivative of this function, but the involvement of matrix operations in the exponential term crosses the threshold into matrix calculus, somewhat beyond my limited mathematics background. I have already made an attempt at this partial derivative, but am not entirely confident that my derivation is correct. Perhaps a second pair of eyes might help - could you tell me whether this derivation is correct, and if not, where I have made a mistake?

Assume I have the following function:

$$g(\mathbf{x})=f(\mathbf{x})\exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x}) \tag{1}$$

where $\mathbf{x}=[x_1,x_2,...,x_D]^T\in\mathbb{R}^D$, $f(\mathbf{x}):\mathbb{R}^D\rightarrow\mathbb{R}$, and $\Sigma$ is a covariance matrix. I wish to take the partial derivative of this expression with regards to a map component $x_i, 1 \leq i \leq D$, i.e.:

$$\frac{\partial}{\partial x_i}g(\mathbf{x})=\frac{\partial}{\partial x_i}\left(f(\mathbf{x})\exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x})\right) \tag{2}$$

Applying the chain rule yields:

$$\frac{\partial}{\partial x_i}g(\mathbf{x})=\frac{\partial f(\mathbf{x})}{\partial x_i} \exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x}) + f(\mathbf{x}) \frac{\partial \exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x})}{\partial x_i} \tag{3}$$

The first summand is complete, but we can expand the second summand further. Using the derivative rule for exponential functions $exp(f(x))'=exp(f(x))f'(x)$ we obtain:

$$\frac{\partial}{\partial x_i}g(\mathbf{x})=\frac{\partial f(\mathbf{x})}{\partial x_i} \exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x}) + f(\mathbf{x}) \exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x}) \frac{\partial -\mathbf{x}^T\Sigma^{-1}\mathbf{x}}{\partial x_i} \tag{4}$$

Here I am not entirely sure... treating $\frac{\partial}{\partial x_i}$ as a left-side matrix operator, it only affects the left-most $\mathbf{x}^T$. If we have:

$$\frac{\partial \mathbf{x}^T}{\partial x_i}=[0,...,1_i,...0]^T$$

where $1_i$ denotes a $1$ in the $i$-th component of the vector, then we have:

$$\frac{\partial}{\partial x_i}g(\mathbf{x})=\frac{\partial f(\mathbf{x})}{\partial x_i} \exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x}) - f(\mathbf{x}) \exp(-\mathbf{x}^T\Sigma^{-1}\mathbf{x}) [0,...,1_i,...0]^T\Sigma^{-1}\mathbf{x} \tag{5}$$

which can be simplified to:

$$\frac{\partial}{\partial x_i}g(\mathbf{x})= \exp(\mathbf{x}^T\Sigma^{-1}\mathbf{x}) \left( \frac{\partial f(\mathbf{x})}{\partial x_i} - f(\mathbf{x}) [0,...,1_i,...0]^T\Sigma^{-1}\mathbf{x} \right) \tag{6}$$

Is this derivation correct?


$ \def\p{{\partial}} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\i#1{#1^{-1}} $For typing convenience define the matrix variable $$M=\Sigma^{-1}$$ and use a colon as a convenient product notation for the trace, i.e. $\;A:B = {\rm Tr}(AB^T)$

Then take the logarithms of the functions before differentiating $$\eqalign{ \log g &= \log f - M:xx^T \\ \i{g}dg &= \i{f}df - M:\left(dx\,x^T+x\,dx^T\right) \\ dg &= \frac gf\left(\grad{f}{x}\right):dx - g\left(M+M^T\right)x:dx \\ \grad{g}{x} &= \frac gf\left(\grad{f}{x}\right) - 2gMx \\ }$$ This agrees with your result, except that you are missing a factor of two on the rightmost term.
So you made a mistake differentiating the $\left(x^T\Sigma^{-1}x\right)$ quadratic form.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.