# Why are inner products in RKHS linear evaluation functionals?

I'd like to know why inner products in Reproducing kernel Hilbert spaces are (linear) evaluation functionals.

I understand that inner products are linear functionals, and I know what an evaluation functional is, I just can't explain why an inner product (in a RKHS) is evaluation functional, and vise-versa.

-

On a Hilbert space, all continuous linear functionals are inner product functionals (Riesz), and conversely (Cauchy-Schwarz). In a RKHS, evaluation functionals are continuous, which is equivalent to being inner product functionals. The converse is usually not true. That is, inner product functionals on a RKHS need not be point evaluations.

Let $H$ be a Hilbert space consisting of complex-valued functions on a set $X$ such that for each $x\in X$, the evaluation functional $f\mapsto f(x)$ is continuous. Then for each $x\in X$ there is a $k_x\in H$ such that for all $f\in H$, $f(x)=\langle f,k_x\rangle$. (The function $K:X\times X\to \mathbb{C}$ defined by $K(x,y)=k_y(x)=\langle k_y,k_x\rangle$ is the reproducing kernel of the RKHS $H$, and some authors start with $K$ in defining a RKHS.) Only the inner products with the elements $k_x$ are evaluation functionals.

For example, consider the Hardy space $H^2$ of holomorphic functions on the open unit disk whose sequences of Maclaurin series coefficients are in $\ell^2$, with inner product $\displaystyle{\left\langle \sum_{k=0}^\infty a_kz^k,\sum_{k=0}^\infty b_kz^k\right\rangle=\sum_{k=0}^\infty a_k\overline{b_k}}$. Evaluations on the open disk are continuous, as can be seen directly by writing down the element $k_w$ of $H^2$ whose inner product functional is evaluation at $w$, $k_w(z)=\sum_{k\geq0} \overline{w}^kz^k=\frac{1}{1-\overline{w}z}$. So a necessary and sufficient condition for an inner product functional $f\mapsto\langle f,g\rangle$ to be an evaluation functional is the existence of a $w$ in the open unit disk such that $g=k_w$, a condition which typical $g\in H^2$ will not satisfy. Note that the set of evaluation functionals is not closed under scalar multiplication, nor addition. In fact, it is linearly independent.

A simpler but in some ways less interesting example is $\ell^2$ thought of as a space of functions on the nonnegative integers, where the evaluation functionals are just the inner products with elements of $\ell^2$ that have the value $1$ at one point and vanish elsewhere. An even simpler example would be a finite dimensional Hilbert space thought of as functions on a finite set. In these cases, cardinality is enough to see that most inner product functionals are not point evaluations.

The inner product functionals and evaluation functionals would be identical if you were considering a Hilbert space $H$ as a space of functions on its dual space, in the usual isomorphism of $H$ with its double dual.

Added: Here is some elaboration on the first 2 sentences. If $H$ is a Hilbert space and $g\in H$, then the function $T_g :H\to \mathbb{C}$ defined by $T_g(f)=\langle f,g\rangle$ is a linear functional on $H$ called an "inner product functional" above. Each inner product functional is continuous. The operator norm of $T_g$ is equal to $\|g\|$. The Cauchy-Schwarz inequality gives $|T_g(f)|\leq \|f\|\|g\|$ for all $f$, which means $\|T_g\|\leq\|g\|$. Plugging $g$ into $T_g$ gives $|T_g(g)|=\|g\|^2$, showing that $\|T_g\|\geq \|g\|$.

So inner product functionals are continuous, and this would be true in any inner product space. The Riesz representation theorem (for Hilbert space, sometimes also called Riesz's lemma) says that the converse is true for a Hilbert space. You can see this for example in the Wikipedia article, and in many textbooks including the basics of Hilbert spaces, such as Rudin's Real and complex analysis. That is, if $T:H\to\mathbb{C}$ is any continuous linear functional, then there is a $g\in H$ such that $T=T_g$.

Hopefully the first sentence is clearer now. As for the second sentence, it follows directly from the first sentence and the definition of RKHS, and the second paragraph elaborates on this. There is more than one way to characterize RKHS, and if continuity of point evaluations isn't clear from your definition, perhaps you could provide the definition to make it easier to answer your questions.

-
Thanks for your reply. You've obviously put a lot of thought into it. However, its the first two sentences that I'm struggling with i.e. that "On a Hilbert space, all continuous linear functionals are inner product functionals." And that "In a RKHS, evaluation functionals are continuous, which is equivalent to being inner product functionals." I just can't make these inferences from my current understanding of the Cauchy-Schwartz and Riesz theorems. –  Olumide Feb 20 '11 at 16:33
Also can I add that what I hope to do is to make an intuitive argument for RKHS'. Currently I've found several references that claim that the Hilbert space $\mathcal{L}_2$ is "too big" and contains too many non-smooth functions and also that its evaluation functionals (still don't know how you get those) are unbounded. I assume that RKHS solve both problems and would appreciate help showing that this is indeed the case. –  Olumide Feb 20 '11 at 16:42
After some more surfing the internet I found a document and another that explains the rationale for excluding discontinuous functions. Also, contrary to what I originally thought, a RKHS has the property that at each $x \in \mathcal{R}^n$ there exists an evaluation functional $K_x$ called the representer such that $f(x) \; = \; \langle K_x \; , \; f \rangle$ for any function $f$. –  Olumide Feb 20 '11 at 18:13
... In other words, $K_x$ is the evaluation functional not the inner product. Although it appears that, like a the dirac delta, the $K_x$ is defined in terms of its effect on other functions via the inner product. What remains is to prove that evaluation functionals indeed have this property. (Now I realize that your post covers much of I've just learned.) –  Olumide Feb 20 '11 at 18:20
@Olumide: I elaborated a little after seeing your first 2 comments, and before seeing your last 2. In your second comment, what is $\mathcal{L}_2$? If it is what I think it is, it is not even a space of functions, let alone a RKHS. In your third comment, yes, that is in my post. In your 4th comment, $K_x$ is neither an evaluation nor an inner product; it is an element of your space whose inner product functional is equal to evaluation at $x$. Proving that evaluations have this property (existence of such $K_x$) is direct from definition/Riesz. –  Jonas Meyer Feb 20 '11 at 18:34