5
$\begingroup$

Sometimes I encounter a book or a research paper where the author insists on working with Banach spaces instead of Hilbert spaces.

I am curious as to what would be a non-trivial difference between these two setups for optimization related applications.

For example, you can still define a Frechet derivative on Banach spaces. That's fine. And the majority of optimization concepts such as strong convexity only involves the norm, and not the inner product. Convergence of a sequence of points $x(k)$ towards the optimum $x^\star$ also involves the norm.

Something trivial that can be done in Hilbert space but not Banach space would be taking the inner product (obvious). But even the inner product can be simply written as the sum of the product of the coordinates of a vector. Hence that can be skipped also.

Is there really any significant difference between workng with Hilbert versus Banach spaces?

$\endgroup$
  • 5
    $\begingroup$ 'But even the inner product can be simply written as the sum of the product of the coordinates of a vector.' - this is not the case for infinite dimensional spaces, or at least, needs some refinement (e.g. what is coordinate in a general space?) $\endgroup$ – Berci Jun 30 '19 at 20:13
  • 3
    $\begingroup$ I am not an expert in optimization. What comes to my mind is that the convenient thing about Hilbert spaces is that they allow you to find a unique minimizer for the distance function to a closed subspace (in general orthogonal projection is a handy thing). $\endgroup$ – Severin Schraven Jun 30 '19 at 20:14
  • 2
    $\begingroup$ In Hilbert space the derivative gives an obvious descent direction that does not necessarily have a counterpart in general Banach spaces. $\endgroup$ – copper.hat Jun 30 '19 at 21:04
  • 3
    $\begingroup$ Maybe this is too trivial, but here's an "optimization" problem: take a linear functional and see if its norm is attained. In a Hilbert space, it is. In a Banach space, not necessarily. $\endgroup$ – Nate Eldredge Jul 1 '19 at 7:06
  • $\begingroup$ @NateEldredge Could you elaborate as an answer? $\endgroup$ – The man of your dream Jul 1 '19 at 23:36
6
$\begingroup$

Here is a nice result which can be proven in Hilbert and Banach spaces, but the assumptions hide the fact that the space is already Hilbert:

Let $X$ be a Banach space, $f \colon X \to \mathbb R$ be twice Fréchet differentiable in a given point $\bar x \in X$ and suppose that there exists $\alpha > 0$ such that $$f'(\bar x) = 0, \qquad f''(\bar x)[h,h] \ge \alpha \, \|h\|_X^2 \;\forall h \in X.$$ Then, for all $\varepsilon > 0$ there is $\delta > 0$ such that $$f(x) \ge f(\bar x) + \frac{\alpha-\varepsilon}{2}\,\|x - \bar x\|^2 \quad\forall x \in X, \|x-\bar x\|_X \le \delta.$$

I hope this counts as a non-trivial result. In infinite-dimensional optimization, this theorem is quite useful, since it implies some stability of the minimizer $\bar x$ w.r.t. perturbations of the problem (i.e. a discretization can be seen as a perturbation).

The proof uses just a second-order Taylor expansion of $f$ at $\bar x$ and does not need an inner product or a Hilbert space structure.

However, it can be easily checked that $$ (g,h) \mapsto f''(\bar x) [g,h] $$ defines an inner product on $X$ and the associated norm is equivalent to $\|\cdot\|_X$. Hence, $X$ has to be (isomorphic) to a Hilbert space and the theorem is not applicable in Banach spaces.

$\endgroup$
3
$\begingroup$

There are closed forms for nearest point projections in Hilbert space that are implementable: e.g., hyperplane and unit ball.

In "nice" Banach spaces, projections do exist but there are typically no formulas available. This is really no fun when dealing with projection or proximal mappings even in finite-dimensional settings. So optimization in Banach spaces is much harder if you want implementable algorithms.

$\endgroup$
1
$\begingroup$

In Hilbert spaces the squared norm is Frechet differentiable, in particular $$ \frac{d}{dx} || x ||^2 = 2x, $$ which is not true for Banach spaces. Smoothness is a very important property for optimization - not just for methods (gradient descent) but also for optimality conditions ($\nabla f(x) = 0$).

$\endgroup$
  • $\begingroup$ Why is it not true for Banach spaces? It's not obvious to me $\endgroup$ – The man of your dream Jul 12 '19 at 5:14
  • $\begingroup$ I don't have an easy answer, but the most obvious difference is that in my statement about the derivative I implicitly identified the Hilbert space with its dual which can't be done in Banach spaces. $\endgroup$ – xel Jul 12 '19 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.