Meaning of product-like construction in Hilbert space

Question

In [1, Assumption 2] the expression $$|\nabla^3 \ell_z(\theta)[k,h,h]|\leq \sup_{g\in\mathcal{H}}|k\cdot g| \nabla^2\ell_z(\theta)[h,h] $$ is used, where $\ell_z:\mathcal{H}\to \mathbb{R}$ with a reproducing kernel Hilbert space $\mathcal{H}$.

I don't understand the meaning of $[k,h,h]$ and cannot find a definition in the paper. Do I miss some standard construction in Hilbert spaces?

[1] http://proceedings.mlr.press/v99/marteau-ferey19a/marteau-ferey19a.pdf

Answer 1

Here, $\ell$ (I drop the suffix $z$ for convenience) is defined on some Hilbert space $\mathcal H$ which has no reason to be $\mathbb R^d$. Hence its gradient $\nabla\ell$, Hessian $\nabla ^2\ell$, and all higher order derivatives $\nabla ^n\ell$ (assuming they exist) need to be defined a little bit differently than in the Euclidean case.

• Let's start with the gradient $\nabla \ell$ : the direct analogue of the gradient for functions defined on Hilbert spaces would be the Fréchet derivative. For $\ell : \mathcal H \to \mathbb R$ and $\theta\in\mathcal H$, we define the Fréchet derivative of $\ell$ at $\theta$ as the unique bounded linear functional $\nabla \ell(\theta) : \mathcal H \to \mathbb R$ such that, $$\lim_{h\in\mathcal H,\|h\|\to 0}\frac{\|\ell(\theta+h) - \ell(\theta) - \nabla\ell(\theta)[h]\|}{\|h\|} = 0$$ Hence you see that for any point $\theta\in\mathcal H$, $\nabla\ell(\theta) $ is a linear functional defined on $\mathcal H$ rather than a vector. However, Riesz representation theorem tells us that for any bounded linear functional $f$ on $\mathcal H$, there exists a unique $f^*\in\mathcal H$ such that $f(x) = \langle f^*,x\rangle $ for all $x\in \mathcal H$. In the special case $\mathcal H = \mathbb R^d$, this tells us that the Fréchet derivative $Df(x_0)$ of a differentiable map $f:\mathbb R^d \to \mathbb R$ at a point $x_0\in \mathbb R^d$ is given by $Df(x_0) : x \mapsto \langle \nabla f(x_0),x \rangle$, where $\nabla f(x_0)$ is the usual gradient of $f$ at $x_0$. All that is to say that, indeed, the Fréchet derivative extends the definition of gradient to general Hilbert spaces (normed spaces, actually).

• If you get the first definition, then it is not hard to extend it to define the Hessian $\nabla^2\ell$ : For some parameter $\theta\in\mathcal H$ and $\ell :\mathcal H \to \mathbb R$, we define the Hessian of $\ell$ at $\theta$ as the Fréchet derivative of $\nabla \ell (\theta) $, i.e. the unique bounded linear operator $\nabla ^2\ell(\theta) : \mathcal H \to \mathscr L(\mathcal H,\mathbb R)$ such that $$ \lim_{h\in\mathcal H,\|h\|\to 0}\frac{\|\nabla\ell(\theta+h) - \nabla\ell(\theta) - \nabla^2\ell(\theta)[h]\|}{\|h\|} = 0$$ Notice in particular that $\nabla^2\ell(\theta)[h]$ is not a real number but rather a bounded linear functional defined on $\mathcal H$. This implies by Riesz theorem that given a fixed $\theta \in \mathcal H$, for all $h\in\mathcal H$, there exists a unique element $\text{Hess}(\theta)[h]\in\mathcal H $ such that $$ \nabla ^2\ell(\theta)[h][x] \equiv \nabla ^2\ell(\theta)[h,x] = \langle \text{Hess}(\theta)[h], x \rangle$$ This implies that $\nabla^2\ell(\theta)$ can be uniquely identified as a bilinear map defined on $\mathcal H \times \mathcal H$

• If you understand the above then it's not hard to see (but tedious to write) how to generalize the above definitions to higher order derivatives of $\ell$, and you should be able to convince yourself that for any integer $n\ge 1$, $\nabla^n\ell$ is a real-valued multilinear map defined on the product space $\mathcal H\times \ldots \times \mathcal H$.

That was a lot, but if you followed you should now be able to answer your question : In the equation you write, the terms in bracket $[k,h,h]$ and $[h,h]$ do not represent a "product", but they are rather the respective arguments of the multilinear maps $\nabla^3\ell_z(\theta) $ and $\nabla^2\ell_z(\theta)$, defined on $\mathcal H \times \mathcal H \times\mathcal H $ and $\mathcal H \times\mathcal H$ respectively.