5
$\begingroup$

I need a help with one example. I have to proove that hessian matrix of a quadratic form $f(x)=x^TAx$ is $f^{\prime\prime}(x) = A + A^T$. I am not even sure how the Jaxobian looks like (I never did one for $x \in R^n$). Thanks for any advice.

$\endgroup$
7
$\begingroup$

So let's compute the first derivative, by definition we need to find $f'(x)\colon\mathbb R^n \to \mathbb R^n$ such that $$ f(x+h) = f(x) + f'(x)h + o(h), \qquad h \to 0 $$ We have \begin{align*} f(x+h) &= (x+h)^tA(x+h)\\ &= x^tAx + h^tAx + x^tAh + h^tAh\\ &= f(x) + x^t(A + A^t)h + h^tAh \end{align*} As $|h^tAh|\le \|A\||h|^2 = o(h)$, we have $f'(x) = x^t(A + A^t)$ for each $x \in \mathbb R^n$. Now compute $f''$, we have \begin{align*} f'(x+h) &= x^t(A + A^t) + h^t(A + A^t)\\ &= f(x) + h^t(A + A^t) \end{align*} So $f''(x) = A + A^t$.

$\endgroup$
  • $\begingroup$ I don't understand how in the last step we can make $x^t(A+A^t)=f(x)$? $\endgroup$ – user 42493 Oct 5 at 15:04
4
$\begingroup$

Intuitively, the gradient and Hessian of $f$ satisfy \begin{equation} f(x + \Delta x) \approx f(x) + \nabla f(x)^T \Delta x + \frac12 \Delta x^T Hf(x) \Delta x \end{equation} and the Hessian is symmetric.

In this problem, \begin{align*} f(x + \Delta x) &= (x + \Delta x)^T A (x + \Delta x) \\ &= x^T A x + \Delta x^T A x + x^T A \Delta x + \Delta x^T A \Delta x \\ &= x^T A x + \Delta x^T(A + A^T)x + \frac12 \Delta x(A + A^T) \Delta x. \end{align*}

Comparing this with the approximate equality above, we see that $\nabla f(x) = (A + A^T) x$ and $Hf(x) = A + A^T$.

$\endgroup$
3
$\begingroup$

Write explicitly $$f(x)=\sum_{i,j}(\text{2nd degree monomials})$$ The hessian is the matrix $$H=(\partial_i\partial_jf(x)).$$

$\endgroup$
2
$\begingroup$

For $f(x)=x^{\top}Ax$ where $f(x)\colon\mathbb R^n \to \mathbb R^1$, the Jacobian $f'(x)\colon\mathbb R^n \to \mathbb R^n$ can be solved as

$f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h}$

$f(x+h)=(x+h)^{\top}A(x+h)=(x^{\top}A+h^{\top}A)(x+h)=x^{\top}Ax+x^{\top}Ah+h^{\top}Ax+h^{\top}Ah$

$f(x+h)=f(x)+x^{\top}Ah+x^{\top}A^{\top}h+h^{\top}Ah=f(x)+x^{\top}(A+A^{\top})h+h^{\top}Ah$

$f'(x)=\lim_{h\to0}\frac{f(x)+x^{\top}(A+A^{\top})h+h^{\top}Ah-f(x)}{h}=\lim_{h\to0}\frac{(x^{\top}(A+A^{\top})+h^{\top}A)h}{h}$

$f'(x)=\lim_{h\to0}x^{\top}(A+A^{\top})+h^{\top}A=x^{\top}(A+A^{\top})$

Thus, the Hessian $f''(x)\colon\mathbb R^n \to \mathbb R^{n\times n}$ can be found as

$f''(x)=\lim_{h\to0}\frac{f'(x+h)-f'(x)}{h}$

$f'(x+h)=(x+h)^{\top}(A+A^{\top})=x^{\top}(A+A^{\top})+h^{\top}(A+A^{\top})$

$f''(x)=\lim_{h\to0}\frac{x^{\top}(A+A^{\top})+h^{\top}(A+A^{\top})-x^{\top}(A+A^{\top})}{h}=\lim_{h\to0}A+A^{\top}$

Finally $f''(x)=A+A^{\top}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.