1
$\begingroup$

I answered this question, but I'd like to understand more details about the matrix notation behind it (and that's why I'm making another post). We have $f:\Bbb R^n\to \Bbb R$ given by $$f(\theta) \doteq \alpha e^{-\beta \theta^\top\theta}, $$alright. We want to compute the bilinear map ${\rm Hess} f (\theta)$. Since I recognize $g (\theta)\doteq\theta^\top \theta$ as $\langle \theta,\theta\rangle$ (of course, $\langle \cdot,\cdot\rangle$ denotes the usual scalar product), I see that $$Dg(\theta) = 2\langle \theta, \cdot \rangle = 2\theta^\top, $$and hence $\nabla g (\theta) = 2\theta $. Then chain rule gives $$\nabla f (\theta) = -2\alpha \beta e^{-\beta \theta^\top \theta}\theta $$as the OP of the linked question states, so far so good.

I'm having trouble doing something similar to check that $${\rm Hess}f (\theta)=2\alpha \beta e^{-\beta \theta^\top\theta}(2\beta \color{blue}{\theta\theta^\top}-{\rm Id}_n).$$I do not want to use components as I did there.

A simple attempt is to use the product rule together with ${\rm d}\theta ={\rm Id}_n $. Differentiating the expression for $\nabla f (\theta) $ we get $$-2\alpha\beta (e^{-\beta\theta^\top\theta}(-2\beta \theta^\top)\theta +e^{-\beta \theta^\top\theta}{\rm Id}_n) = 2\alpha \beta e^{-\beta \theta^\top\theta}(2\beta\color{red}{\theta^\top\theta}-{\rm Id}_n), $$but this doesn't compile, and I can't see why the order comes out wrong.

So I'd like to know exactly what identification am I missing here. I also recognize $\theta\theta^\top$ as the matrix of the bilinear map $\theta \otimes \theta$, and I'm comfortable with tensor products, so you can come in with guns blazing, if needed.

Thanks.

$\endgroup$
1
$\begingroup$

Although you've already used $g$, I'd like to use it to denote the gradient, i.e. $\,\,g=\nabla f$

Find the differential of the gradient, then the hessian $$\eqalign{ g &= -2\beta f\theta \cr dg &= -2\beta(\theta\,df+f\,d\theta) \cr &= -2\beta(\theta g^Td\theta+fI\,d\theta) \cr H=\frac{\partial g}{\partial\theta} &= -2\beta\,(\theta g^T+fI) \cr &= 2\beta\,\,\big(\theta(2\beta f\theta)^T-fI\big) \cr &= 2f\beta\,\,(2\beta\,\theta\theta^T-I) \cr }$$ As expected, this is your result but with the change $$\theta^T\theta \implies \theta\theta^T$$

$\endgroup$
  • $\begingroup$ So basically I screwed up the order in the product rule. This clarifies it, thanks! $\endgroup$ – Ivo Terek Oct 3 '17 at 16:37
1
$\begingroup$

I don't know exactly what you mean by "using matrix notation efficiently", since there is no matrix on the post (:P), but I think that it is something along the following lines.

As you have concluded, $g(x):=\nabla f(x)=-2\alpha \beta e^{-\beta\langle x,x \rangle}x$. By the product rule, $$(\mathrm{Hess}_x f)(h)=g'_x(h)=-2\alpha \beta e^{-\beta\langle x,x\rangle}h+2\alpha \beta e^{-\beta \langle x,x\rangle}2\beta\langle x,h \rangle x$$ $$=2\alpha \beta e^{-\beta \langle x, x\rangle}(2\beta\langle x,h\rangle x-h). $$ So we have to understand the linear maps $$h \mapsto 2\beta \langle x, h \rangle x$$ and $$h \mapsto h.$$ The latter is obvious: it is $\mathrm{Id}$. The former is precisely $2\beta x^* \otimes x$, where $x^*=\langle x, \cdot\rangle$, under the canonical identification $Hom(V;W) \leftrightarrow V^*\otimes W.$ Matricially, note that $(x^* \otimes x)(e_i)=x^*(e_i)x=x_ix,$ which says precisely that $x^* \otimes x=x x^T$ (of course, you could do this computation without using the fact that the map can be represented by $2\beta x^* \otimes x$, but I think this makes things clearer).

$\endgroup$
  • $\begingroup$ This is really helpful, thanks. Just one question: the hessian shouldn't be quadratic in $h$ instead of linear? I'm missing the identification again. (And by the way, I literally tossed a coin to choose which answer to accept, but life goes on) $\endgroup$ – Ivo Terek Oct 3 '17 at 16:39
  • 1
    $\begingroup$ @IvoTerek If you see the hessian as the derivative of $\nabla f: \mathbb{R}^n \to \mathbb{R}^n$, it will come out (at a point) as a linear map $Hess_x: \mathbb{R}^n \to \mathbb{R}^n$, as I did above. If you compute the Hessian as the second derivative of $f$, it will come out as a linear map $\mathbb{R}^n \to L(\mathbb{R}^n;\mathbb{R})$, which is identifiable to a bilinear map (i.e, an element of $L(\mathbb{R}^n,\mathbb{R}^n;\mathbb{R})$), which is what you are referring to in your comment. (...) $\endgroup$ – Aloizio Macedo Oct 3 '17 at 17:25
  • 1
    $\begingroup$ (...) More explicitly, the underlying identifications among all this are the identifications between: (1) Bilinear symmetric maps to $\mathbb{R}$, (2) Symmetric matrices, (3) "special" (i.e., "symmetric") Linear maps to the dual (or to the space, using the identification given by inner product - which in practice, in this situation, transforms the map $\mathbb{R}^n \to L(\mathbb{R}^n,\mathbb{R})$ to a map $\mathbb{R}^n \to \mathbb{R}^n$ by means of the gradient),(4) Quadratic Forms. The computation I gave is using (3) in the form of linear maps to the space. $\endgroup$ – Aloizio Macedo Oct 3 '17 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.