# Is my statistician friend right/wrong on metric spaces and norms?

I was talking to a statistician friend of mine who said that instead of minimizing this function $\sum_{i,j}W_{ij}d_{ij}^2(X)$ over $X$ it would be better to solve an analogous minimization problem $\sum_{i,j}W_{ij}||f(Y_{i.})-f(Y_{j.})||_\mathcal{H}$ over $f$ instead with the norm being a RKHS(Reproducing Kernel Hilbert Space) norm where the functions $f(.)$ come from a Hilbert space of functions. Both minimizations are under a constraint that $\sum_{i,j}d^2_{i,j}(X)$ or $\sum_{i,j}||f(Y_{i.})-f(Y_{j.})||_\mathcal{H}$ is constrained to a fixed real positive value $\nu$.

Here, $d_{ij}^2(X)$ is the squared Euclidean distance between the rows $i,j$ of the real unknown matrix $X$. $Y$ is a fixed real matrix.

Question: Why would the latter problem under the function space be better!? What does the RKHS norm provide, which is different than the euclidean distance/norm? From this description, why do you think my academic friend has said this-when thought from different mathematical directions? What are we gaining or losing between these two formulations? After all, once the $f(.)$ is solved for, the norms seem to be preserving some sort of notion of distance/dissimilarity or nearness. What is special about the second problem!? I understand that the question is a little open ended. So please feel free to be verbose in expressing your thoughts!

Perspective: I showed him the initial Euclidean problem. He thought about it and posed the other problem as being interesting. Please shed your thoughts.

-
Why is no one answering! But just voting up? ;) I'd rather have more discussion and no votes ;) –  halms Mar 28 '13 at 19:24
My experience is that model builders tend to use Hilbert space norms.This is something I, myself, am still learning about. I'm not sure why this is a better approach. You might take a look at this reference. amazon.com/… –  mtiano Mar 28 '13 at 19:29
An equivalent way to pose your question would be: "What does a Hilbert space buy me that I don't get from my standard Euclidean space?" I have a few ideas as to why, but I'm not really knowledgeable enough to not sound like I'm talking out of my rear-end. –  Arkamis Mar 28 '13 at 19:41
@Arkamis..That is exactly my question, that you have put forth succinctly. I put in a big description just to make sure that I cannot maintain the standard of abstraction required for say a usual problem that i would post in se. Please feel free to putforth some of your thoughts/pointers to material-as I do need a starting point-from some direction or the other! –  halms Mar 28 '13 at 20:12

This is not an answer, per se, but some general observations.

What you're looking to do is similar to a curve fit problem or a regression problem: you have some data, and you'd like to find some function that fits this data in some minimizing sense. Because you're using the difference between observations, it's obviously not quite as straightforward, but there is some similarity here.

Statisticians and data analysts often view data as the effect of some cause. While some are thrilled to model the effect (e.g. obtaining a function that fits the data, and saying "a-ha! The effect looks like this!"), a more interesting problem is to try to infer some structure that underlies the source data -- understanding the mechanics of the cause. Of course, you're not really understanding the cause without some extrinsic information, but that's how we like to think of it. Some measurements $y$ are the result of some physical process $f$ acting on some inputs $x$.

There are many, many ways to do this, many of which are more or less equivalent. As @mtiano mentioned, model builders love Hilbert space norms.

What a Hilbert space norm buys you, in this case, is a notion that you can find some function that models the cause, and then in doing so, you get a model of the effect for free.

In other words, instead of interpolating a bunch of output data, you're finding a function that operates on some fixed input data; once you have that function, you can do all the things you would do with an interpolant, but now you can directly link that interpolant to some independent variables.

Depending on your data analysis approach, there are many ways to approach this problem. Things like neural networks, adaptive parameter estimation algorithms, quasi-Newton search methods, etc. tend to default to $L^2$-minimization. I'd wager that in engineering and statistics papers, $L^2$ is by far the most common "optimization space."

$L^2$, of course, is a Hilbert space. And we like Hilbert spaces because they have inner products, and inner products are things that tend to arise naturally in many of these analyses.

Of course, you don't have to work in $L^2$ if it doesn't suit your application. But all the work that's been done in developing algorithmic and analytical techniques for optimization and data analysis in $L^2$ are going to translate almost immediately to some other Hilbert space.

Not having the inner product means that you might have to do some additional work, and in the end, all you get is an interpolant on some observed data, with no intuition regarding the underlying process that made the data look like it does.

-
One thing to note: I said that many techniques for data analysis are more or less equivalent. In reality, many of them are essentially the same. Neural networks, radial basis function networks, nonlinear regression, fuzzy networks, nonlinear parameter identification, principle component analysis, proper orthogonal decomposition, and so on and so on are all essentially the same thing. –  Arkamis Mar 29 '13 at 3:49
If you do any reading on any of these techniques, the minimization problem usually becomes one of minimizing a residual in an $L^2$ sense. Hence, by working in a Hilbert space for your problem, you're essentially almost transforming your problem to one already solved. Almost. –  Arkamis Mar 29 '13 at 3:50