Let $\mathbf{Y} \in \mathbb{R}^n$ be a column vector, $\boldsymbol{\mu} \in \mathbb{R}^n$, and $A$ be a $n\times n$ matrix of constants not dependent on $\mathbf{Y}$.

Definition. $\boldsymbol{\Sigma} = AA^{T}$, and we assume $\boldsymbol{\Sigma}$ is positive definite (comes immediately from $A$ being invertible).

Calculate $\left|\dfrac{\text{d}}{\text{d}\mathbf{Y}}\left[A^{-1}\left(\mathbf{Y}-\boldsymbol{\mu}\right)\right]\right|$ with respect to $|\boldsymbol{\Sigma}|$.

Problem. I haven't actually been given a definition of $\dfrac{\text{d}}{\text{d}\mathbf{Y}}\left[A^{-1}\left(\mathbf{Y}-\boldsymbol{\mu}\right)\right]$ for this problem, other than that it is the "matrix of partial derivatives" (no further explanation beyond that). For those of you who are familiar with multivariate statistics, this is used in the derivation of the PDF of the multivariate normal distribution.

Approach #1. Use computational formula that I found in a different book: $$\dfrac{\text{d}}{\text{d}\mathbf{Y}}\left[A^{-1}\left(\mathbf{Y}-\boldsymbol{\mu}\right)\right] = A^{-1}$$ and then using that $A^{-1} = A^{T}\boldsymbol{\Sigma}^{-1}$: $$|A^{-1}| = |A^{T}||\boldsymbol{\Sigma}^{-1}| \Longleftrightarrow \left(|A^{-1}|\right)^2=|\boldsymbol{\Sigma}|^{-1} \implies |A^{-1}|=|\boldsymbol{\Sigma}|^{-1/2}\text{,}$$ which is exactly what I want.

Approach #2. Compute the matrix of partial derivatives elementwise, and take the determinant.

I have no idea how to do it this way, and I think it's the way that my text wants me to approach the problem. So I think it should look something like this:

$$\dfrac{\text{d}}{\text{d}\mathbf{Y}}\left[A^{-1}\left(\mathbf{Y}-\boldsymbol{\mu}\right)\right] = \begin{bmatrix} \dfrac{\partial [a_1 (y_1 - \mu_1)]}{\partial y_1} & \cdots & \dfrac{\partial [a_1 (y_n - \mu_n)]}{\partial y_n} \\ \vdots & \vdots & \vdots \\ \dfrac{\partial [a_n (y_1 - \mu_1)]}{\partial y_1} & \cdots & \dfrac{\partial [a_n (y_n - \mu_n)]}{\partial y_n} \end{bmatrix}$$ where $A^{-1}=[a_{ij}]$, $\boldsymbol{\mu}=[\mu_i]$ and $\mathbf{Y} = [Y_i]$.

On second thought, this doesn't seem right because I think the elements of the resulting matrix should be sums.

How do I do this problem using approach #2?


You almost have it.

(In case you don't know about it, I'll use here Einstein's convention of summation over repeated index. That is, a sum like $\sum_ka^i_kb^k$ will be written as simply $a^i_kb^k$.)

Call the vector $A^{-1}({\bf Y} - \boldsymbol{\mu})\equiv {\bf F}({\bf Y})$. ${\bf F}$ is a function mapping vectors into vectors, i.e., ${\bf F}:\mathbb{R}^n\rightarrow \mathbb{R}^n$. Its gradient is a matrix $\boldsymbol{\nabla}{\bf F}$ given by $$(\boldsymbol{\nabla}{\bf F})^{ij}\,\equiv\,\frac{\partial {\bf F}^i}{\partial {\bf Y}^j}\,=\,\frac{\partial [(A^{-1})^i_k\,(y^k-\mu^k)]}{\partial y^j}$$.

Using your notation, the first row is $$\frac{\partial[a^1_k(y^k-\mu^k)]}{\partial y^1},\cdots,\frac{\partial[a^1_k(y^k-\mu^k)]}{\partial y^n}$$ and the last row is $$\frac{\partial[a^n_k(y^k-\mu^k)]}{\partial y^1},\cdots,\frac{\partial[a^n_k(y^k-\mu^k)]}{\partial y^n}$$.

Clearly, $(\boldsymbol{\nabla}{\bf F})^{ij}\,=\,(A^{-1})^i_j$, which is the result you got above.

Note: If you want to explicitly show the summation symbol, the gradient is $$(\boldsymbol{\nabla}{\bf F})^{ij}\,=\,\sum_k\frac{\partial [(A^{-1})^i_k\,(y^k-\mu^k)]}{\partial y^j}$$

  • $\begingroup$ Thank you very much. Is there a text that you would recommend that goes through this material? $\endgroup$ – Clarinetist Sep 8 '15 at 1:46
  • $\begingroup$ Maybe any textbox in advanced calculus. I haven't checked one of those in years. The wikipedia page on the gradient I link to below doesn't provide a good reference -at least not to my liking. I don't think they would offer a smooth intro to this topic. $\endgroup$ – MASL Sep 8 '15 at 2:32

It is sometimes worth it to go back to the original definition of the (Gateaux) derivative. This avoids messy partials or passing to a particular basis; the derivative is a linear map that is independent of the coordinate system you use. If $f(Y)$ is a vector-valued function taking vector variable $Y$, then for another vector $V$ (with the same dimensions as $Y$), the directional derivative $df/dY$ at $V$ is $$ \frac{df(Y)}{dY}(V) = \lim_{h \rightarrow 0} \frac{f(Y + hV) - f(Y)}{h}. $$ In your case $f(Y) = A^{-1}(Y - \mu)$, so if you work out the above limit you'll get the result you wanted.

  • $\begingroup$ Thank you very much. I've never heard of the Gateaux derivative. I am mainly only familiar with multivariable undergrad real analysis - where would I have to go from here to learn the material? $\endgroup$ – Clarinetist Sep 8 '15 at 2:02
  • $\begingroup$ There is a lot more to the story, but at least if you're only dealing with continuously differentiable functions on finitely-many variables (like in your question), all the notions you're used to are equivalent: gradient, matrix derivative, Gateaux derivative, they are all the same. They begin to differ if your function is not smooth, however. I was just pointing out that you can go back to the original limit definition of derivative to compute these things sometimes. If you look at the limit I wrote out, it's just the difference quotient definition that you should have seen many times. $\endgroup$ – Christopher A. Wong Sep 8 '15 at 2:25
  • $\begingroup$ @Christopher Just to add some precision to the discussion: I didn't mean to give a precise definition of gradient, but it so happens that the gradient is actually implicitly defined as the function ${\bf \nabla}{\bf F}$, s.t., $\lim_{\|{\bf h}\|\to 0}\frac{\|{\bf F}({\bf x}+{\bf h})-{\bf F}({\bf x})-{\bf \nabla}{\bf F}\cdot{\bf h}\|}{\|h\|}=0$. See en.wikipedia.org/wiki/Gradient#Gradient_as_a_derivative $\endgroup$ – MASL Sep 8 '15 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.