So I have gotten stumped on something that seems like it (should?) be easy. I am trying to find the following derivative shown below. I have scoured the wiki link on matrix derivatives, and I think my answer is correct but I want to make sure.

So let us say we have a square matrix $\boldsymbol{A}$, and a vector $\boldsymbol{\theta}$. (I am assuming here that the dimensionality is 2 for ease. So:

$$\boldsymbol{A} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} &a_{22}\end{bmatrix}, \boldsymbol{\theta}= \begin{bmatrix} \theta_0 \\ \theta_1\end{bmatrix}$$

I am trying to derive how we get:

$$ \frac{\delta \boldsymbol{A}\boldsymbol{\theta}}{\delta \boldsymbol{\theta}}= \boldsymbol{A} $$

So first I tried to 'open up' the matrix-vector product, so I now have the following matrix:

$$ \begin{bmatrix} a_{11}\theta_{0} + a_{12}\theta_{1} \\ a_{21}\theta_{0} + a_{22}\theta_{1} \end{bmatrix}_{2x1} $$

... and this is where I am stuck. How do I show from here, that the derivative of the above is indeed equal to $\boldsymbol{A}$? I know that I have to take the partials, but I cannot seem to find a rule governing in what ordering of columns/rows those partials must be taken.

  • $\begingroup$ What you're looking for is called the Jacobian matrix. You should be able to apply the definition given there to get the expected answer. $\endgroup$ – user856 May 8 '12 at 2:12

The derivative is the 'best' linear approximation to a function at a given point. More explicitly, if $f$ is differentiable at a point $x$, then there is a linear approximation (that is, the derivative) $L$ such that $$ f(x+h) -f(x) = Lh + o(h).$$ You can view $h$ as a perturbation, and $Lh$ is the 'best' linear approximation to the corresponding change in $f$. (See https://en.wikipedia.org/wiki/Big_O_notation#Little-o_notation for a description of $o(h)$, it is roughly a way of expressing a limit without using the limit sign.)

There are two points to this ramble:

The first is that if $f$ is linear, then it is it's own derivative. You need do no more work. In your case, the derivative of the function $\theta \mapsto A \theta$ must be $A$.

The second is that this view can help you sort out the whole row/column/order thing. Let $f$ denote your function $\theta \mapsto A \theta$, that is $f(\theta) = A \theta$. If you 'perturb' the first variable ($\theta_0$ in your notation) by an amount $\delta$, then we have $f(\theta + \binom{\delta}{0})-f(\theta) = A \binom{\delta}{0} = \delta \binom{a_{11}}{a_{21}}$. So you can see that the 'linear response' to perturbing $\theta_0$ by $\delta$ is given by the first column of $A$, so we have $\binom{a_{11}}{a_{21}} = \binom{\frac{\partial f_1(\theta)}{\partial \theta_0}}{\frac{\partial f_2(\theta)}{\partial \theta_0}}$.

In general, the $n$th column of the derivative corresponds to perturbing the $n$th variable.

More explicitly, in your case, the derivative is given by $$\begin{bmatrix} \frac{\partial f_1(\theta)}{\partial \theta_0} & \frac{\partial f_1(\theta)}{\partial \theta_1} \\ \frac{\partial f_2(\theta)}{\partial \theta_0} &\frac{\partial f_2(\theta)}{\partial \theta_1}\end{bmatrix} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} &a_{22}\end{bmatrix} .$$

  • $\begingroup$ Thank you copperhat, I am slighly confused as to the last part, should it not be $ \begin{pmatrix} \frac{\delta f_{1}(\theta}{\delta\theta_{0}} \\ \frac{\delta f_{2}(\theta}{\delta\theta_{0}} \end{pmatrix} $ ? $\endgroup$ – Spacey May 8 '12 at 15:25
  • $\begingroup$ @Mohammad: My apologies, I made a mistake. I have fixed it. Thanks for catching it! $\endgroup$ – copper.hat May 8 '12 at 15:46
  • $\begingroup$ Ok. I get this now... I think... So can any and all of the derivatives shown in my wiki link be derived in this manner? $\endgroup$ – Spacey May 8 '12 at 16:50
  • $\begingroup$ They can, but I would urge you to think in terms of what the derivative means in a coordinate-free sense, then the usual coordinate-based approach will follow easily (I think!). $\endgroup$ – copper.hat May 8 '12 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.