Given $g:R^n \rightarrow R^k$ and $h:R^k \rightarrow R$, we have $f(x) = h(g(x))$.

Using the chain rule, we can differentiate $f(x)$ to get

$f'(x) = \nabla^Th(g(x))g'(x)$

My question is why do we take the transpose of the gradient of $h$? Is it just to make sure the result is a scalar, since $f(x)$ is in $R$?

If so, does it mean that every time we do vector differentiation, we need to ensure the output matches the size of the result, and take transpose if necessary (i.e. no hard and fast rule of taking transpose)?

  • $\begingroup$ Edit your post and include the exact definitions of $'$ and $\nabla$. $\endgroup$ – Martín-Blas Pérez Pinilla Feb 5 '16 at 17:51
  • $\begingroup$ I don't understand what you mean. $'$ is derivative, $\nabla$ is gradient... $\endgroup$ – Rayne Feb 5 '16 at 18:06
  • $\begingroup$ I asked this question because of Slide 18 in this set of slides. (see.stanford.edu/materials/lsocoee364a/03convexfunctions.pdf) In the proof for $f''(x)$, I had to find $f'(x)$ first and I was wondering why there was a need to take transpose for $\nabla h$. That's all the exact definitions of $h$ and $g$ that I know. $\endgroup$ – Rayne Feb 5 '16 at 18:19
  • $\begingroup$ I'm asking row or column? Usually, if $h$ is a scalar function, $\nabla h(x)=(h′(x))^T$ with $\nabla h(x)$ column vector and $h′(x)$ row vector. But the important fact is that your result $f′(x)$ can't be a scalar if $n>1$. $\endgroup$ – Martín-Blas Pérez Pinilla Feb 5 '16 at 18:21
  • $\begingroup$ In the slides the usual convention about rows and columns is inverted and the gradient is a row vector. In any case, if $n>1$, $f'(x)$ can't be a scalar. $\endgroup$ – Martín-Blas Pérez Pinilla Feb 5 '16 at 18:30

First, the result isn't an scalar, is a (row) vector. Second, lousy notation. The usual formula for the chain rule is $$D(h\circ g)(x) = Dh(g(x))Dg(x)$$ where the product in the RHS is the matrix product. In your case $\nabla^T h(g(x))$ (i.e., $Dh(g(x))$) is a row vector. See https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant.

EDIT: example with $n = 3$, $k = 2$: $$ \pmatrix{\partial_1 f&\partial_3 f&\partial_3 f} = \pmatrix{\partial_1 h&\partial_2 h} \pmatrix{\partial_1 g_1&\partial_2 g_1&\partial_3 g_1\cr\partial_1 g_2&\partial_2 g_2&\partial_3 g_2}. $$

  • $\begingroup$ Ok, the result is a 1-by-1 vector. Actually, in my case, $g'(x)$ is a column vector, so the result is obtained by multiplying a row vector (after taking the transpose) and a column vector. My question still stands, do we take the transpose to get a 1x1 result? $\endgroup$ – Rayne Feb 5 '16 at 17:14
  • $\begingroup$ @Rayne, isn't The result is a vector with $n$ components. $\endgroup$ – Martín-Blas Pérez Pinilla Feb 5 '16 at 17:16
  • $\begingroup$ $g'(x)$ is a k-by-1 column vector. My lecturer mentioned the result is multiplying a row vector with a column vector. $\endgroup$ – Rayne Feb 5 '16 at 17:23
  • $\begingroup$ @Rayne, are you sure that $g:R^n \rightarrow R^k$? $g'(x)$ can't be a vector if $n>1$. $\endgroup$ – Martín-Blas Pérez Pinilla Feb 5 '16 at 17:43
  • $\begingroup$ @Rayne, see my edit. I've mixed the names of the functions in my original post, $\endgroup$ – Martín-Blas Pérez Pinilla Feb 5 '16 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.