# Vector as whole differentiated has different result compared to element-wise

I'm working with bigger matrices and there differentiation/multiplication. I also want to solve the problem by hand and moved from element-wise expressions to whole components since this makes computation by hand more compact. A Quaternion $q=[q_w \ q_x \ q_y \ q_z]^T$ for instance becomes $q = [w \ v]$. $w$ for the real part and $v$ for the imaginary components. Now I do have an expression like:

$\frac{\partial (v^Tv) \ a}{ \partial v}$

With $v,a \in \mathbb{R}^{3\times1}$

Now, if I process this element-wise and calculate the Jacobian I get:

$x(a,v)=\begin{bmatrix} a_1 (v_1^2 + v_2^2 + v_3^2) \\ a_2 (v_1^2 + v_2^2 + v_3^2) \\ a_3 (v_1^2 + v_2^2 + v_3^2) \end{bmatrix} = \begin{bmatrix} (a_1 v_1^2 + a_1 v_2^2 + a_1 v_3^2) \\ (a_2 v_1^2 + a_2 v_2^2 + a_2 v_3^2) \\ (a_3 v_1^2 + a_3 v_2^2 + a_3 v_3^2) \end{bmatrix}$

$\frac{\partial x(a,v)}{\partial v} = 2 \begin{bmatrix} a_1 v_1 & a_1 v_2 & a_1 v_3 \\ a_2 v_1 & a_2 v_2 & a_2 v_3 \\ a_3 v_1 & a_3 v_2 & a_3 v_3 \end{bmatrix} = 2 \ a \ v^T$

But if I differentiate it based on the non-element-wise expression and apply the product rule I get:

$\frac{\partial (v^Tv) \ a}{ \partial v} = \frac{\partial v^T}{ \partial v} \ v \ a + v^T \ \frac{\partial v}{ \partial v} \ a + v^T \ v \frac{\partial a}{ \partial v} = \mathbf{I}^T \ v \ a + v^T \ \mathbf{I} \ a + 0 = 2 \ v^T \ a$

The first result $(2 \ a \ v^T)$ is a 3x3 Matrix and the second $(2 \ a \ v^T)$ is a scalar.

Since I expect a 3x3 matrix, the element-wise method seems to deliver the right result. The question is, what went wrong with the product rule applied to the non-element-wise expression?

If doing this "non-element-wise", you have to recognize that the vector terms don't necessarily automatically commute, and include the appropriate effects. For example, when using the product rule, you need to make sure the particular term you are differentiating is moved to the right hand side before actually taking the derivative. Thus, for your middle term, since $\left( {{{\underline {\bf{v}} }^T}\underline {\bf{v}} } \right)\underline {\bf{a}} = \underline {\bf{a}} \left( {{{\underline {\bf{v}} }^T}\underline {\bf{v}} } \right)$, the proper expression is $$\underline {\bf{a}} \,{\underline {\bf{v}} ^T}\frac{{\partial \underline {\bf{v}} }}{{\partial \underline {\bf{v}} }} = \underline {\bf{a}} \,{\underline {\bf{v}} ^T}\underline {\overline {\bf{I}} } = \underline {\bf{a}} \,{\underline {\bf{v}} ^T}$$ The first term is a little more subtle, since it involves the derivative of ${\underline {\bf{v}} ^T}$, but it ends up providing the same contribution as the middle term, which is why you actually get the same answer you do doing it "element-wise". While this specific result should be fairly intuitive, one very formal way of seeing what is going on is to use the very important identity for vectorizing matrix products $${\rm{vec}}\left( {\underline {\overline {\bf{A}} } \,\underline {\overline {\bf{B}} } \,\underline {\overline {\bf{C}} } } \right) = \left( {{{\underline {\overline {\bf{C}} } }^T} \otimes \underline {\overline {\bf{A}} } } \right){\rm{vec}}\left( {\underline {\overline {\bf{B}} } } \right)$$ (see the Wikipedia page on Vectorization), where $\otimes$ is the Kronecker product. Note that for matrices (and vectors are just rank 1 matrices, so this still holds) $$\frac{{\partial \underline {\overline {\bf{Y}} } }}{{\partial \underline {\overline {\bf{X}} } }} = \frac{{\partial \,{\rm{vec}}\left( {\underline {\overline {\bf{Y}} } } \right)}}{{\partial \,{\rm{vec}}\left( {\underline {\overline {\bf{X}} } } \right)}}$$
Then, preparing the first term $${\rm{vec}}\left( {\left( {{{\underline {\bf{v}} }^T}\underline {\bf{v}} } \right)\underline {\bf{a}} } \right) = {\rm{vec}}\left( {\underline {\bf{a}} \left( {{{\underline {\bf{v}} }^T}\underline {\bf{v}} } \right)} \right) = \left( {{{\underline {\bf{v}} }^T} \otimes \underline {\bf{a}} } \right){\rm{vec}}\left( {{{\underline {\bf{v}} }^T}} \right) = \underline {\bf{a}} \,{\underline {\bf{v}} ^T}{\rm{vec}}\left( {{{\underline {\bf{v}} }^T}} \right)$$ so that the first term contribution is $$\underline {\bf{a}} \,{\underline {\bf{v}} ^T}\frac{{\partial {\rm{vec}}\left( {{{\underline {\bf{v}} }^T}} \right)}}{{\partial \underline {\bf{v}} }} = \underline {\bf{a}} \,{\underline {\bf{v}} ^T}\frac{{\partial \underline {\bf{v}} }}{{\partial \underline {\bf{v}} }} = \underline {\bf{a}} \,{\underline {\bf{v}} ^T}\underline {\overline {\bf{I}} } = \underline {\bf{a}} \,{\underline {\bf{v}} ^T}$$