I defined an Euclidean norm function with a smooth technique to avoid not differentiable problem in point $0$, $f: \mathbb{R}^n\to\mathbb{R}$, $$f(x) = \sqrt{\|x\|^2+\epsilon^2},$$ where $\epsilon=0.00001$.

I know for sure that $f(x)$ is convex, so $\nabla^2 f(x) \ \forall x\in \mathbb{R}^n$ positive semi-definite.

My question: is the Hessian matrix $\nabla^2 f(x) \ \forall x\in \mathbb{R}^n$ positive definite?

I scaled down the problem to 1-dimension, such that we can write down the gradient(first derivative) and the Hessian( derivative) easily, $$f'(x) = \frac{x}{\sqrt{(x^2+\epsilon^2)}}$$ $$f''(x) = \frac{\epsilon^2}{(x^2+\epsilon^2)\sqrt{(x^2+\epsilon^2)}}.$$

It follows $f''(x)>0 \ \forall x\in\mathbb{R}$, which make it a positive definite matrix.

Can we generalize it to high dimensions $n$?

  • $\begingroup$ You can calculate $\nabla^2 f(x) = (\lVert x \rVert^2 + \varepsilon^2)^{-\frac{1}{2}}I - xx^\top(\lVert x \rVert^2 + \varepsilon^2)^{-\frac{3}{2}}$ $\endgroup$
    – Meowdog
    Jul 29 at 22:12
  • $\begingroup$ If you use the euclidean norm, then for $x=(x_1,...,x_n)^T$, we have $f(x)= \left\|\pmatrix{x_1\\\vdots\\ x_{n}\\\epsilon}\right\|$. Now use $\|x\|= \sqrt{x^T x}$. Calculate $\nabla^2 f$ and substitute in $\epsilon$. Notice that you get a sum of a rescaled identity matrix and a rank-1-matrix. Use the Matrix determinant lemma to calculate the determinant of all relevant minors. $\endgroup$
    – Sudix
    Jul 29 at 22:44

$ \def\e{\varepsilon} \def\l{\left} \def\r{\right} \def\p{\partial} \def\lr#1{\l(#1\r)} \def\trace#1{\operatorname{Tr}\lr{#1}} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\hess#1#2#3{\frac{\p^2 #1}{\p #2\,\p #3^T}} $Let $x$ be a vector in ${\mathbb R}^{n}$.
Square the function before calculating the differential and gradient of $f$ $$\eqalign{ f^2 &= x^Tx + \e^2 \\ 2f\,df &= 2x^Tdx \\ df &= \lr{\frac{x}{f}}^Tdx \\ \grad{f}{x} &= \frac xf \;=\; g &\quad\quad\big({\rm Gradient\,vector}\big) \\\\ }$$ Now calculate the differential and gradient of $g$ $$\eqalign{ dg &= \frac{f\,dx-x\,df}{f^2} \\ &= \frac{fI\,dx-x\lr{\frac xf}^T\,dx}{f^2} \\ &= \lr{\frac{f^2I-xx^T}{f^3} }dx \\ \grad{g}{x} &= \frac{f^2I-xx^T}{f^3} \;=\; H &\qquad\big({\rm Hessian\,matrix}\big) \\ }$$ Note that for vectors orthogonal to $x$, the action of the Hessian matrix is simple linear scaling $$Hv = {\frac{v}{f}} \qquad\qquad\qquad\qquad\qquad $$

Vector components which are parallel to $x$ gets are scaled more aggressively. In the limiting case $$Hx = {\frac{\e^2x}{f^3}} \qquad\qquad\qquad\qquad\qquad $$

So setting $\,\e^2=0\;$ would make $Hx=0,\;$ and the matrix would be semi-definite.

However, since $\e^2>0\;$ the matrix is positive-definite.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.