So for the derivation for linear regression in matrix form, the prof here says we need to use the result that $\frac{\partial A \theta}{\partial \theta}=A^T$ in order to derive the formula. I'm trying to show that this is true but I don't seem to be getting the right answer.

Suppose we have \begin{align} \vec{x}= \left[ \begin{matrix} x_1 \\ x_2 \end{matrix} \right], A= \left[ \begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} \right] \end{align}

Then, \begin{align} A\vec{x}= \left[ \begin{matrix} a_{11}x_1 + a_{12}x_2 \\ a_{21}x_1 + a_{22}x_2 \end{matrix} \right] \end{align}

From wikipedia, we see that the Jacobian gives us: \begin{align} \frac{\partial A\vec{x}}{\partial \vec{x}}= \left[ \begin{matrix} \frac{\partial (A\vec{x})_1}{\partial x_1} & \frac{\partial (A\vec{x})_1}{\partial x_2}\\ \frac{\partial (A\vec{x})_2}{\partial x_1} & \frac{\partial (A\vec{x})_2}{\partial x_2} \end{matrix} \right] = \left[ \begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} \right] = A \end{align}

This is not the same as $A_T$, does anyone know why?

  • $\begingroup$ Probably because what you are doing bears little resemblance with the differential. How do you define the differential of $F$ at $\theta$, already? $\endgroup$ – Did Jun 8 '16 at 6:13
  • $\begingroup$ Hmm, what do you mean by differential of $F$ at $\theta$? $\endgroup$ – Zoom Bee Jun 8 '16 at 6:19
  • 1
    $\begingroup$ Ha! ... What do you mean by $\frac{\partial A \theta}{\partial \theta}$? $\endgroup$ – Did Jun 8 '16 at 6:22
  • $\begingroup$ It depends which way you define the Jacobian of $u$, as $\partial_i u_j$ or $\partial_j u_i$. Do it backwards and you'll be off by a transpose. $\endgroup$ – u54112 Jun 8 '16 at 6:22

The problem is that notation for derivatives in higher dimensions is extremely poor and generally fraught with confusion: it is not standardized, with different authors writing many different closely-related but subtly-different concepts using a hodgepodge of different and inconsistent notation.

If you have a function $f: \mathbb{R}^m \to \mathbb{R}^n$, you can write down a differential which maps, at every point $x\in \mathbb{R}^m$, tangent vectors $v$ at $x$ to tangent vectors $df(x,v)$ at $f(x)$. This differential can be defined by way of the directional derivative:

$$df(x,v) = \frac{d}{dt} f(x+vt) \Big\vert_{t\to 0}.$$

Since the directional derivative is linear in the direction, it must be that the differenetial at $x$ can be expressed as matrix-vector multiplication (*):

$$df(x,v) = [df(x)]v$$ for some matrix $df(x)$ (which is not always easy to write down, mind), often called the Jacobian.

To further complicate matters one sometimes speaks of the gradient of $f$, especially if $f$ is real-valued, in which case the gradient of $f$ at $x$ is the unique vector $\nabla f(x)$ for which $$\langle \nabla f(x), v\rangle = df(x,v),$$ where the inner product is usually (**) the Euclidean dot product, in which case the gradient is the transpose of the differential. Colloquially people often conflate the gradient and Jacobian, which is wrong and inevitably leads to confusion.

Now when you see notation like $\frac{\partial A\theta}{\partial\theta}$ you have to decipher what the author means: is this the Jacobian of the function $f(\theta) = A\theta$? The gradient? Something else? If it's the Jacboian it's not hard to compute that $$\frac{d}{dt} A(\theta + t v)\Big\vert_{t\to 0} = Av$$ and so $[df(x)] = A$, which is in agreement with your calculation. To get $A^T$ the author must mean the notation to represent some type of gradient, although I suspect that the matrix in the video is symmetric, in which case it doesn't much matter whether you have $A$ or $A^T$.

(*): Complications arise once you try to define the differential of matrix-valued or -dependent functions, which I won't get into here.

(**): Although in physics (where the gradient arises frequently) the inner product is usually not Euclidean but rather given by a mass matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.