What does $\nabla \left(\exp{\{\eta^{T}{\bf{u(x)}}\}}{\bf{u(x)}}\right)=\exp{\{\eta^{T}{\bf{u(x)}}\}}{\bf{u(x)}}{\bf{u(x)}}$ mean?

A couple of related questions:

Suppose we want to calculate the gradient $\nabla_{\eta} (\exp{\{\eta^{T}{\bf{u(x)}}\}})$ (as Muphrid suggested, $\nabla_{\eta}$ means the gradient with respect to the variable $\eta$.) Obviously, we would have something like this

$$\nabla_{\eta} (\exp{\{\eta^{T}{\bf{u(x)}}\}}) = \exp{\{\eta^{T}{\bf{u(x)}}\}}{\bf{u(x)}}$$ where $\eta$ and $u(x)$ are vectors. In the equation above, what operation is implied by the gradient? Inspecting the equation, I guess it is straightforward to assume that since $\eta^{T}{\bf{u(x)}}$ produces a scalar and gradients produce vectors (or higher dimensional quantities), then this operation is unambiguous. But what about the following:

$$\nabla_{\eta} \left(\exp{\{\eta^{T}{\bf{u(x)}}\}}{\bf{u(x)}}\right)=\exp{\{\eta^{T}{\bf{u(x)}}\}}{\bf{u(x)}}{\bf{u(x)}}$$

Applying the same logic as before, ${\bf{u(x)}}{\bf{u(x)}}$ should be a vector or something else, but definitely not a scalar. So the simplest choice is to assume that is a matrix. Therefore, ${\bf{u(x)}}{\bf{u(x)}}$ means ${\bf{u(x)}} {\bf{u(x)}}^{T}$ which as far as I know is correct (and makes sense, a gradient of a vector is a matrix)

But what is a principled way of dealing with this kind of operations?

UPDATE:

I have just seen another example that is a bit more confusing than the previous ones. What should be the result of $\nabla \left(\frac{1}{2}{\bf{w}}^{T}{\bf{w}}\right)$? ${\bf{w}}^{T}$ or $\bf{w}$? According to the answer of Brady Trainor, it should be a contravariant tensor of rank 1 which means the resulting vector is a column vector. Is that right? The minimization of the equation

$$\frac{1}{2}\sum_{n=1}^{N}\{t_{n} - {\bf{w}}^{T}\phi(x_{n})\}^{2}+\frac{\lambda}{2}{\bf{w}}^{T}{\bf{w}}$$

resulting in

$${\bf{w}} = (\lambda I + \Phi^{T}\Phi)^{-1}\Phi^{T}{\bf{t}}$$

leads to me to believe that the correct answer is a row vector ${\bf{w}^{T}}$

-
So $\eta$ is a vector, and $\nabla$ takes the gradient with respect to this vector? – Muphrid Mar 20 '13 at 6:24
@Muphrid Exactly. – Robert Smith Mar 20 '13 at 6:26

Tensor products can be hard to interpret sometimes. There is a related notion call the geometric product of vectors. The geometric product of $a$ and $b$ is denoted $ab$ and equal to $a \cdot b + a \wedge b$, where $a \wedge b$ is a member of the exterior algebra, called a bivector, and interpreted as an oriented plane. Bivectors can be represented with skew-symmetric matrices, while the parts of the dot product can be put on the diagonal. This is a way of getting a matrix out of the geometric product, but for the most part, actually using those matrices is unnecessary.

So we're getting a little ways away from matrix calculus, but we're all covering the same ground.

Let's talk about differentiation. The vector derivative with respect to a vector $\eta$ is called $\nabla_\eta = e^1 \frac{\partial}{\partial \eta^1} + \ldots$. Strictly speaking, $\nabla$ is a covector (a row vector, a cotangent vector, etc) and you have to account for this accordingly. The vector derivative can be combined with vectors and such using the geometric product also, combining divergence and curl into one operation.

Now then, let's look at your first problem. This is made easier using the chain rule. Let $a, b$ be a vector and $f, g$ be vector fields. This is the chain rule:

$$b \cdot \nabla_a (f \circ g)(a) = [b \cdot \nabla_a g(a)] \cdot \nabla_g f(g)$$

We can apply this to our problem at hand.

$$b \cdot \nabla_\eta (u \exp [u \cdot \eta]) =u (b \cdot \nabla_\eta [u \cdot \eta]) \frac{d}{d(u \cdot \eta)} \exp (u \cdot \eta)$$

The result is $u (b \cdot u) \exp (u \cdot \eta)$. Differentiating with $\nabla_b$ gives us the originally desired formula, yielding $(u \cdot u) \exp(\eta \cdot u)$. This is a scalar--the curl parts are identically zero.

Let's look at your second problem. $\frac{1}{2} \nabla_w (w \cdot w)$. We can attack this by symmetry and holding one of the $w$ constant, as $\nabla_w (w \cdot a) = a$ for any constant $a$. This clearly gives $w$, but as a row vector. In a metric space, there's no real difference; you can convert between vectors and covectors freely, but the derivative of a scalar should always give a row vector.

-
Thanks. A couple of questions: In the first problem, you get a scalar. However, the solution is actually a matrix. I suppose the problem comes from the term $b \cdot \nabla_\eta (u \exp [u \cdot \eta])$ in which you're implying there is a dot product between b and the rest of the expression. As far as I know, the result should be a matrix, not a scalar. In the second problem, the rule that says 'the derivative of a scalar should always give a row vector' makes sense but do you mind to explain why is there such rule? – Robert Smith Mar 23 '13 at 5:22
It should be $(b \cdot \nabla_\eta)$ instead; doing a directional derivative first instead of the full derivative is a cheap and easy trick to make invoking the chain rule easier, which I eventually undo in the end by differentiating with $\nabla_b$. Why gradients should be row vectors: think of column vectors as the directions tangents to curves and row vectors as those perpendicular to surfaces. The gradient is like the latter, as it points orthogonal to a level surface of a scalar field. This isn't a coincidence; it's constructed that way. – Muphrid Mar 23 '13 at 5:46
Secondly, why the result is a scalar: you may end up with a matrix answer, yes. That matrix represents the sum of a scalar part (from the dot product, lying on the diagonal) and a bivector part (from the wedge product, the off-diagonal terms). Being symmetric, there is an orthogonal transformation that diagonalizes this matrix, making evident that the bivector part is zero. The trace (the dot product) is invariant, however. Hence, the answer is a "matrix," but a matrix that represents only the dot product, only a scalar. – Muphrid Mar 23 '13 at 5:52
Great. I understood why gradients should be row vectors but you lost me on the second question. Can you recommend a source to learn more about your answer. – Robert Smith Mar 23 '13 at 19:38
@RobertSmith it probably lies a bit outside the strict purview of matrix calculus. Both Geometric Algebra for Physicists and Clifford Algebra to Geometric Calculus have in-depth sections on the calculus of multivectors. You're probably right for this problem to go with the answer being a symmetric matrix; it's just the geometric interpretation of that matrix that I'm emphasizing. – Muphrid Mar 24 '13 at 19:49

EDIT: I think I should have written $e^i{\partial\over\partial x^i}$, rather than $e^i{\partial\over\partial x_i}$

How about $u=u_ie^i=u_1e^1+\cdots+u_ne^n$? Then, we have

$$uu=u_iu_je^ie^j=u_iu_je^{ij},$$

which is like the matrix you were asking about. I think I took a tensor product there, that is, when we write $e^ie^j$, might interpret as $e^i\otimes e^j$, and write as $e^{ij}$. So we can say $u\otimes u$?

In order for it to "take" row vectors on the left and column vectors on the right, you'd have to use an association $e^{ij}\mapsto e_j^i$. The matrix we know and love is written $u_iu^je_j^i$.

For $\nabla$, we write

$$\nabla=e^i\partial_i=e^1{\partial\over\partial x_1}+\cdots+e^n{\partial\over\partial x_n}.$$

Compare $\nabla u$ to $\nabla\cdot u$. Use the fact that $e^ie^j=e^{ij}$, while $e^i\cdot e^j=e_ie^j=\delta_{ij}$, which is $1$ if $i=j$, and $0$ otherwise.

-
That's a good idea, although I would have to use tensor notation in the whole equation to be consistent. Is there a rule that allows unambiguity dealing with matrices? – Robert Smith Mar 21 '13 at 16:25
Hard to say without digging through the internet myself. Can you write what $\nabla_\eta$ means, for reference? Also, are you using the product rule above? Can you share your work? – Brady Trainor Mar 21 '13 at 18:15
Sorry, $\nabla_\eta$ means the gradient with respect to $\eta$ – Robert Smith Mar 21 '13 at 18:18
Then I'll take a random stab at what that means. For a general tensor $T$, $$\nabla_\eta(T)=\nabla T\cdot \eta.$$ Then we have $$(\partial_ie^iT)\cdot \eta_ie^i.$$ The \cdot means we will have to raise and lower indices on the left hand object. Does this look right so far? Does it work with what you have so far? Am I way off? – Brady Trainor Mar 21 '13 at 18:23
Well, not quite because then I would get only $\exp{\{\eta^{T}{\bf{u(x)}}\}}{\bf{u(x)}}\cdot{\bf{u(x)}}$ and that's not correct. Unlike tensor notation, we don't have explicit unit vectors to produce, in this case, the resulting matrix. – Robert Smith Mar 21 '13 at 18:28