2
$\begingroup$

This question already has an answer here:

I would like to derive the following expression with respect to a vector $x \in \mathcal{R}^N$:

$a a^T b$ where $a, b$ are vectors in $\mathcal{R}^M$.

My idea is to apply the following

$ \frac{\partial a}{\partial x} a^T b + a (\frac{\partial a}{\partial x})^T b + a a^T \frac{\partial b}{\partial x}$ but it is dimensionally wrong.

In fact: $\frac{\partial a}{\partial x} a^T b$ has dimension: $(MxN) (NxM) (Mx1)$ or $a (\frac{\partial a}{\partial x})^T b$ has dimension: $(Mx1) (NxM) (Mx1)$ that are really dimensionally wrong.

I'm assuming that the derivative of a vector $a \in \mathcal{R}^M$ with respect to another vector $a \in \mathcal{R}^N$ is a matrix $a \in \mathcal{R}^{MxN}$.

Moreover, even in the very simple example: I want to derive $a^T b$ with respect to $x$: $(\frac{\partial a}{\partial x})^T b + a^T \frac{\partial b}{\partial x}$, the first addendum seems to be a column vector in $\mathcal{R}^N$ while the second one is a row vector in $\mathcal{R}^N$. What am I doing wrong? I know I can fix this by transposing the function $a^T b$ when partially deriving with respect to $a$ (then the first addendum becomes $b^T \frac{\partial a}{\partial x}$) but I would like to know if there is a generic rule for that.

Thank you to anyone that can provide to me some hints.

Marco


My question is a bit different with respect to this post since that is the differentiation about the transpose of a vector while here I'm questioning about the dimensionality of vector product derivative.

Moreover, I add a dummy question that may help me in solving this: In case of a derivative of scalar-vector product, i.e. $\frac{\partial (c a)}{\partial x}$, with $c \in \mathcal{R}, a \in \mathcal{R}^M$ and $x \in \mathcal{R}^N$, I would say that the derivative is $\frac{\partial (c a)}{\partial x} = \frac{\partial c}{\partial x} a + c \frac{\partial a}{\partial x}$. Now, the first addendum has a wrong dimensionality: (1xM) (Nx1). Now, I know that the things works if I do: $\frac{\partial (c a)}{\partial x} = a \frac{\partial c}{\partial x} + c \frac{\partial a}{\partial x}$, i.e., if I set the scalar on the right side before deriving with respect to $c$. In this case I know that, for dimensionality, $a c$ is more correct than doing $c a$ but I would like to know if there are some rules that I'm aware of. Then my question is: why do I have to do this?

$\endgroup$

marked as duplicate by Alex M., Yujie Zha, Namaste, Shailesh, Parcly Taxel May 27 '17 at 2:14

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ My question is a bit different with respect to this post since that is the differentiation about the transpose of a vector while here I'm questioning about the dimensionality of vector product derivative. $\endgroup$ – Neostek May 24 '17 at 12:00
2
$\begingroup$

Note that $$a^Tb=b^Ta$$ since these vectors are real.

With that in mind, let's find the differential of this vector function $$\eqalign{ y &= aa^Tb \cr dy &= da\,a^Tb + a\,da^T\,b + aa^T\,db \cr &= a^Tb\,da + ab^T\,da + aa^T\,db \cr\cr }$$ Now that we've rearranged the expression and moved the differentials in each term to the rightmost position, we can write the gradient as $$\eqalign{ \frac{\partial y}{\partial x} &= (a^Tb)\,\frac{\partial a}{\partial x} + ab^T\,\frac{\partial a}{\partial x} + aa^T\,\frac{\partial b}{\partial x} \cr\cr\cr }$$ As for your "dummy" question, note that $$z = \lambda a = a\lambda$$ since scalar multipliers commute with vectors.

Use the same recipe as last time -- move differentials to the right, find the gradient. $$\eqalign{ dz &= \lambda\,da + a\,d\lambda \cr \frac{\partial z}{\partial x} &= \lambda\,\frac{\partial a}{\partial x} + a\,\frac{\partial\lambda}{\partial x} \cr }$$

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.