Relationship between inner product and norm

Question

I understand that there can be many different types of norms (e.g. mean norm, Cartesian norm, supremum norm etc). Are there also other types of inner products apart from $\langle x,y \rangle= \sum_{j =1}^n x_j y_j$? Also, I read that for any inner product on a vector space V the function $\|x\| = \sqrt{\langle x,x \rangle}$ defines a norm on the vector space. Why is that so? So does this formula work for all the different inner products?

Answer 1

Yes, there are many different types of inner products. Consider the inner product on $L^2$ given by $\langle f, g \rangle = \int f(x) \overline{g(x)} dx$.

An inner product $\langle , \rangle$ always defines a norm by the formula $||x||^2 = \langle x, x \rangle$. You can check that all the conditions of a norm are satisfied. However, the converse is not true, that is, not every norm gives rise to an inner product. Norms which satisfy the parallelogram law can be used to define inner products via the polarization identity.

Answer 2

In $\mathbb{R}^n$ and $\mathbb{C}^n$ we can actually present a characterization of all inner products. Fix an inner product $\langle \cdot , \cdot \rangle$, probably the Euclidean one. Then for any positive definite matrix $P$, $(x,y) = \langle x,Py \rangle$ is also an inner product. Here positive definiteness is defined in terms of $\langle \cdot , \cdot \rangle$.

Conversely, any inner product on a finite dimensional space over $\mathbb{R}$ or $\mathbb{C}$ has such a representation. In particular, given an inner product $(\cdot , \cdot)$, we have $P_{ij}=( e_i,e_j )$, where $\{ e_i \}_{i=1}^n$ is an orthonormal system with respect to $\langle \cdot , \cdot \rangle$. This gives the representation of $P$ in terms of the orthonormal coordinates given by $\langle \cdot , \cdot \rangle$. If the base inner product is the Euclidean one, then $\{ e_i \}_{i=1}^n$ can be taken to be the standard basis, in which case this is the usual representation.

The fact that that every inner product induces a norm is nearly a simple consequence of the definition of an inner product. Specifically, $\| x \| \geq 0$ with $\| x \| = 0$ iff $x = 0$ is built completely into the definition. $\| a x \| = |a| \| x \|$ is also trivial, because $\| a x \| = (ax,ax) = a \overline{a} \| x \|^2$, then $|a| = \sqrt{a \overline{a}}$. The tricky part is the triangle inequality. This is typically proven with the Cauchy-Schwarz inequality.

Answer 3

The weighted sum formula for the inner product, $$(x,y) = \sum_i w_i x_i y_i,$$ captures the inner product's essential meaning - in finite dimensions all inner products act like a weighted sum in some basis (also the idea can be generalized to infinite dimensions using the spectral theorem, the sum basically becoming an integral)

Why? Well by definition inner products are symmetric positive definite bilinear forms, so on finite dimensions they always have a representation by a symmetric positive definite matrix, $$(x,y) = x^T M y.$$

Taking the singular value decomposition $M = U \Sigma U^T$ (or you could do eigenvalue decomposition since it's a SPD matrix), that gives $$(x,y) = x^T U \Sigma U^T y$$

So, if $x_i,y_i$ are the components of $x,y$ in the basis of the singular vectors of $M$ then you can write the inner product in a weighted sum type form $$(x,y) = \sum_i \sigma_i x_i y_i,$$

where the $\sigma_i$'s are the singular values from the diagonal matrix $\Sigma$ .

---

In terms of norms, the unit balls for a norm induced by an inner product are ellipsoids, with axes given by the singular vectors, and axis lengths determined by the singular values. So in a more geometric sense, there is a direct correspondence between ellipsoids and inner products.

Answer 4

Any pair of inner product spaces of equal and finite dimension are indeed isomorphic, so there is no meaningfully different inner product; to demonstrate that, notice that, from any basis $b_1,b_2,\ldots,b_n$ of a vector space, you can, knowing the inner product, remove the components of $b_i$ parallel to $b_1,\ldots,b_{i-1}$, ensuring that it is orthogonal to any of those, and thereby, it must be possible to find a orthogonal basis of unit vectors. A linear map between two spaces of equal dimension, taking one orthogonal basis to another obviously preserves inner products, thus the spaces are isomorphic.

So, there is no inner product, for instance, inducing the norm $||x||=|x_1|+|x_2|$ on $\mathbb{R^2}$ - so the "circle" of vectors where $||v||=1$ is always just a linear transformation of a sphere when the norm is induced by an inner product - and, for finite spaces, it suffices to show that any inner product induces a norm, since a norm is still a norm after application of a linear map.

The only non-trivial thing necessary to show that the inner product space induces a norm is the triangle inequality. Note that, in any space, if we need $||x+y||\leq||x||+||y||$ for particular vectors $x$ and $y$, we just need that, in the span of $x$ and $y$, this holds; however, since the span of $x$ and $y$ is a (at most) two dimensional linear subspace of the inner product space (which is, in itself, a two-dimension inner-product space, obviously). Since the triangle inequality holds for any two-dimensional inner product space (remember that they are all isomorphic), it holds in every inner product space.

Answer 5

For the second question, to give a slightly more abstract point of view, if a space can be equipped with an inner product, as proven by @Ian, it can always be equipped with a norm. In other words, the set of all inner product spaces is a subset of the set of all normed vector spaces. Furthermore, the former is actually a strict subset of the latter since one cannot always induce an inner product given a norm. This "strict subset" relationship generalizes to more "spaces". Namely, the set of all normed vector spaces is a strict subset of the set of all metric spaces, where one can define a distance measure. And the set of all metric spaces is a strict subset of the set of all topological spaces, where one can define some of the most basic mathematical concepts such as convergence.

You might have noticed that some spaces are "better" than others in the sense that they can be equipped with more concepts, which makes it easier to talk about some problems. For example, the concept of orthogonality comes from inner product being zero, but in a topological space, one cannot always define an inner product (since an inner product space is a strict subset of the set of all topological spaces, so when one has a topological space, that space may not be an inner product space), as a result, one may not be able to even talk about orthogonality in that space because it is not defined there. By the way, a Hilbert space (a complete inner product space) is basically THE space with all the nice properties and concepts.

Answer 6

There are many inner products. For instance, take the square real matrices and define $\langle A,B\rangle = tr(AB^T)$ with transpose as conjugation $\langle A,B \rangle = \langle B^T,A^T\rangle$, then $\langle A,B\rangle $ is an inner product.

Going a bit further, from a normed space you can define a metric space by taking $d(x,y) = |x-y|$ (verify that the metric conditions hold). As you pointed out, from an inner product we can define a norm by taking $||x|| = \sqrt{\langle x,x \rangle}$ (also you may verify the norm properties - remember to use Cauchy-Schwarz to prove the triangle inequality).