4
$\begingroup$

I read several descriptions about inner products recently due to an exercise (here it is no longer active so I like to ask again). I am still very confused about this concept. Any clarification will be greatly appreciated.

  1. The very first definition of inner product I encountered is $\langle v, w\rangle =v_1w_1+v_2w_2+v_3w_3$. Or in general, it is a function defined on $\mathbb R^3\times\mathbb R^3$ into $\mathbb R$ which satisfies certain positivity and linearity conditions. I have no problem in understanding this definition.
  2. Then people talk about the set of inner products in $\mathbb R^3$. It seems that an inner product can be represented by a matrix. Why? And how? This does not make sense to me because if we write inner product in vector notation, then we have $\langle v, w\rangle=v^Tw$. There is no matrix in this notation, is there? Moreover, what does it mean by a set of inner products? Does that mean a set of functions (since inner product can be treated as a function as indicated in 1)?
  3. What is the natural topology on the set of inner products? Can I say that the topology is the sub product topology of $\mathbb R^3\times \mathbb R^3$?
$\endgroup$
  • $\begingroup$ You can put a topology on the "space" of inner products? This is news to me. $\endgroup$ – Alex G. Apr 30 '14 at 12:57
  • 1
    $\begingroup$ @AlexGrounds Sure you can. For example, if $V$ is a topological vector space, you can endow $V^2 = V\times V$ with the product topology, and $(V^2)^*$ with e.g. the weak topology. Since inner products on $V$ are linear functionals on $(V^2)^*$, that puts a topology on the space of inner products, no? $\endgroup$ – fgp Apr 30 '14 at 13:18
  • $\begingroup$ Oh, that's cool. Hadn't seen it before. $\endgroup$ – Alex G. May 1 '14 at 3:40
10
$\begingroup$

The space of general inner products on $\mathbb{R}^n$

On $\mathbb{R}^n$, the most general definition of an inner product is that an inner product is some bilinear map $F \,:\, \mathbb{R}^n\times \mathbb{R}^n \to \mathbb{R}$. It's easy to see that the bilinearity implies that $F$ is fully determined by the images of pairs of basis vectors of $\mathbb{R}^n$, i.e. if $(b_i)_{1\leq i \leq n}$ is a basis of $\mathbb{R}^n$, then $F$ is fully determined by the scalars $$ f_{i,j} = F(b_i,b_j) \text{, $1 \leq i,j \leq n$.} $$

It's equally straight-forward to see that every such choice of scalars defines an inner product, because we can always set $$ F(x,y) = \sum_{i=1}^n \sum_{j=1}^n f_{i,j} x_i y_j \quad\text{where } x = \sum_{i=1}^n x_i b_i,\, y = \sum_{j=1}^n y_j b_j \text{.} $$ (i.e. $(x_i)$ respectively $(y_i)$ is the coordinatization of $x$ respectively $y$ in the basis $(b_i)$) This also shows that inner products correspond to $n\times n$ matrices, because if we interpret the $f_{i,j}$ as the coefficients in such a matrix, we get $$ F(x,y) = x^T Ay \text{.} $$

Thus, the inner products on $\mathbb{R}^n$ form a vector space isomorphic to $\mathbb{R}^{n^2}$. If you place additional restrictions on inner products, such as symmetry (i.e., $F(x,y) = F(y,x)$), positive definiteness (i.e. $F(x,x) > 0$ if $x \neq 0$), the resulting space is some subset of $\mathbb{R}^{n^2}$. For some restrictions (like symmetry), you get a subspace, while others (like positive definitness) yield more complex forms of subsets. Which is why your teacher said that you get some manifold.

There are multiple ways to define a topology on the vector space of inner products. Since it's isomorphic to $\mathbb{R}^{n^2}$, one way is to simply use the usual euclidean norm. Another is the define a kind of operator norm, i.e. set $$ \|F\|_{\textrm{op}} = \limsup_{x,y \in \mathbb{R}^n} \frac{|F(x,y)|}{\|x\|\cdot \|y\|} \text{.} $$ By using the representation of $F$ as $f_{i,j}$ for some basis $(b_i)$ of $\mathbb{R}^n$ from above, you can show that an equivalent (though not necessarily identical, I think) norm can be defined as $$ \|F\|_\infty = \max_{1 \leq i,j \leq n} \left|f_{i,j}\right| $$

The subspace $S_n$ of symmetric inner products on $\mathbb{R}^n$

If we fix a basis $(b_i)$ of $\mathbb{R}^n$, symmetry of an inner product $F$ means for the corresponding matrix $A$ that $x^T Ay = y^T A x$. By using that $(uv)^T = v^Tu^T$ for arbitrary $k\times l$ matrices $u$ and arbitrary $l\times m$ matrices $v$, we get $$ y^T A y \overset!= x^T A y = \left(x^T A y\right)^T = (Ay)^T x = y^T A^T y \Rightarrow A = A^T $$ i.e. that $A$ must be a symmetric matrix (how nice that the two meanings of symmetric for matrices and inner products coincide! No risk of confusion there!). It's easy to see that the set $S_n$ of symmetric $n \times n$ matrices form a subspace of $\mathbb{R}^{n^2}$ with $$ \dim S_n = n + \frac{n^2 - n}{2} = \frac{n(n+1)}{2} \text{.} $$ Just observe that a symmetric matrix is fully determined by the coefficient $f_{i,j}$ with $i \leq j$, and that any such choice of coefficients can be extended to a symmetric matrix by setting $f_{i,j} = f_{j,i}$ for $i > j$.

The set $S_n^+$ of symmetric positive definite inner products on $\mathbb{R}^n$

Now we further restrict $S_n$ to the set of positive definite and symmetric matrices $S_n^+$, i.e. to $$ S_n^+ = \{A \in S_n \,:\, x^TA x > 0 \text{ for all $0 \neq x \in \mathbb{R}^n$}\} \text{.} $$ If $A,B \in S_n^+$, then for $C = A+B$ and $x \neq 0$ we have $$ x^T C x = x^T (A+B)x = x^T(Ax + Bx) = \underbrace{x^TAx}_{> 0} + \underbrace{x^TBx}_{> 0} > 0 $$ so $S_n^+$ is closed under sums. Similarly, if $A \in S_n^+$, $x \neq 0$ and $0 < \lambda \in \mathbb{R}$, then for $C = \lambda A$ we have $$ x^T C x = x^T (\lambda A) x = \underbrace{\lambda}_{> 0}(\underbrace{x^TAx}_{> 0}) > 0 \text{.} $$ But if $\lambda = 0$, then $\lambda A = 0 \neq S_n^+$, and for $0 > \lambda \in \mathbb{R} $ we have $x^T\lambda A x < 0$ for $x \neq 0$. $S_n^+$ is therefore not closed under scalar multiplication. So, unfortunately, $S_n^+$ is not a subspace of $S_n$.

The set $S_n^+$ as a manifold

So what, then, can we say about $S_n^+$?. Let's again pick some $A \in S_n^+$, and let $$ \mu_A := \inf_{\|x\| = 1} x^T A x \text{.} $$ Since the unit sphere $\{x \in \mathbb{R}^n \,:\, \|x\|=1\}$ is a compact subset of $\mathbb{R}^n$, and since the map $x \mapsto x^T Ax$ is continuous, $\mu_A > 0$. (Because if not, pick a convergent sequence $x_n \to x$ with $\|x_n\|=1$ and $x_n^TAx_n < \frac{1}{n}$, and use the continuity to show that $x^TAx = 0$. The compactness of the unit sphere plays a crucial role here!).

We now take some arbitrary symmetric matrix $B \in S_n$, and look at $C = A + \lambda B$. As a sum of symmetric matrices, $C$ is obviously symmetric. Let $0 \neq x \in \mathbb{R}^n$ be an arbitrary vector, let $l = \|x\| > 0$, and let $x_1 = \frac{x}{l}$. In other words, $x_1$ is the unique vector on the unit sphere with $x = l x_1 $. Then $$ x^T C x = l^2\left(x_1^T(A + \lambda B)x_1\right) = l^2\big(\underbrace{x_1^TAx_1}_{\geq \mu_A} + \lambda \underbrace{x^TBx}_{\in[-M_B,M_B]}\big) \geq l^2\left(\mu_A - \lambda M_B\right) > 0 \text{ if $\lambda < \frac{\mu_A}{M_B}$} $$ where $$ M_B = \sup_{\|x\|=1} |x^TBx|. \quad \text{($M_B < \infty$, again because $x \mapsto x^TBx$ is continuous)} $$

This is very interesting! It means that we we start from some element in $S_n^+$, we can add scaled versions of arbitrary symmetric matrices $B$ and the result is still positive definite, so long as we keep the scaling factor small enough. We can strenghten this result a bit more. Since $S_n$ is finite-dimensional, we can pick some basis $(B_i)$ of $S_n$, and set $$ M = \max_{1 \leq i \leq \dim S_n} M_{B_i} \quad\text{i.e.}\quad M \leq M_{B_i} \text{ for all basis matrices $B_i$.} $$

It the follows that if $A \in S_n^+$, all the matrices $$ A + \sum_{i=1}^{\dim S_n} \mu_i B_i \quad\text{ where } \sum_{i=1}^{\dim S_n} |\mu_i| < \frac{\mu_A}{M} $$ are positive definite!. This shows that locally, $S_n^+$ has the same dimension as $S_n$, because if we restrict ourselves to some small neighbourhood of a particular symmetric and positive definite $A$, that neighbourhood "looks" just like $S_n$.

The case $\mathbb{R}^2$

In the case of $\mathbb{R}^2$, $\dim S_n = 3$. For $S \in S_2^+$, we must have for all $x_1,x_2$ where one value is non-zero that $$ (x_1,x_2) A (x_1,x_2)^T = a_{11} x_1^2 + a_{22} x_2^2 + 2a_{12} x_1x_2 > 0 \text{.} $$ The cases $(x_1,x_2) = (1,0)$ respectively $(x_1,x_2) = (0,1)$ show that this requires $a_{11} > 0$, $a_{22} > 0$. Rewriting as $$ a_{11} x_1^2 + a_{22} x_2^2 + 2a_{12} x_1x_2 = \left(\sqrt{a_{11}}x_1 + \sqrt{a_{22}}x_2\right)^2 - 2\sqrt{a_{11}a_{22}}x_1x_2 + 2a_{12}x_1x_2 $$ yields for $a_{12}$ the condition $$ |a_{12}| < \sqrt{a_{11}a_{12}} \text{.} $$

We change the base of $S_3$ such that the new coordinate $x,y,z$ obey $x + y = a_{11}$, $x - y = a_{22}$ and $z = a_{12}$, then transformed conditions on $x,y,z$ are $$\begin{eqnarray} x &>& 0 \\ |y| &<& x \\ |z| &<& \sqrt{x^2 - y^2} \text{.} \end{eqnarray}$$ The set $S_3^+$ is thus a cone which is rotationally symmetric around the ray $x > 0, y=z=0$, which in the original coordinates $a_{11},a_{12},a_{12}$ means the ray $a_{11} = a_{22} = c$.

$\endgroup$
  • $\begingroup$ Thank you for your response. Could you be more specific on how to get a sub-manifold and its dimension? Some of my thought is as follows. By your argument, the set of inner products in $\mathbb R^3$ can be represented by a set of $3\times 3$ real matrices denoted here by $(a_{ij})$. Then we need to add conditions for symmetry which amounts to $a_{11}=a_{22}=a_{33}$ and $a_{ij}=a_{ji}$. Then we add conditions for positive definiteness, i.e. $\forall x\neq 0\in \mathbb R^3, x^TAx>0$. Then these equations should define a sub-manifold of $\mathbb R^9$, i.e. either a level set or a graph. Right? $\endgroup$ – LaTeXFan Apr 30 '14 at 13:54
  • $\begingroup$ @20824 OK, I've vastly expanded my answer by adding a proof that locally, $S_n^+$ and $S_n$ have the same dimension. $\endgroup$ – fgp Apr 30 '14 at 15:12
6
$\begingroup$

With regards to your second point, one can actually view the usual inner product $\langle v, w \rangle = v^T w$ as a special case of the more general $\langle v, w \rangle = v^T A w$ where $A$ is a matrix satisfying sufficient conditions to make this an inner product. The usual inner product is just this with $A = I$.

$\endgroup$
3
$\begingroup$

Abstractly, inner products are special cases of something from geometry called a symmetric bilinear form on a vector space. A bilinear form on and $F$ vector space $V$ is just a function $B:V\times V\to F$ satisfying

  • $B(x,y)=B(y,x)$
  • $B(x+y,z)=B(x,z)+B(y,z)$
  • $B(\lambda x,y)=B( x,\lambda y)=\lambda B(x,y)$

You'll notice the positive definiteness criterion ($B(x,x)>0$ for all $x\neq 0$) is missing.

Another thing that's missing is any mention of coordinates or components of vectors! If you pick any basis at all, then you can bulid a matrix $X$, necessarily a symmetric matrix, such that $\vec{x}^\top X \vec{y}=B(x,y)$, where $\vec{x}$ is the coordinate representation of $x$ in that particular basis.

When you are working in the simple basis with $(1,0,0)$, $(0,1,0)$ and $(0,0,1)$ and the usual inner product, then $X$ is just the identity matrix. But keep in mind $X$ doesn't have to be the identity matrix.

Now a natural question is this:

If I'm willing to undergo a change of basis, is there any way I can simplify $X$?

For the case of $F=\Bbb R$, you can always change basis so that the matrix for the bilinear form is zero everywhere except on the diagonal, where it can have $0$'s, $1$'s and/or $-1$'s. If you additionally require that $B(x,x)>0$ for all $x\neq 0$, then $X$ can be changed into the identity matrix. You can learn a little about this from the wiki article on metric signatures.

Also keep in mind that here are some fields $F$ that are not as nice as $\Bbb R$, and won't admit such nice simplifications.


So how can you view the set of inner products (for, say, $\Bbb R$)? For a fixed basis, you can view the set of possible symmetric bilinear forms on $\Bbb R^n$ as the set of $n\times n$ symmetric matrices. If you want to focus on positive definite forms only, then you have to further restrict this to the set of positive definite symmetric matrices.

Any positive definite symmetric matrix at all will define an inner product on the vectors in the basis you chose, and you can see by playing around with some examples that they give different forms. For example, you could take the matrix $X=\begin{bmatrix}1&0\\0&2\end{bmatrix}$ and examine $\vec{v}^\top=(0,1)$. Notice that $\vec{v}^\top X\vec{v}=2 $, but $\vec{v}^\top I_2 \vec{v}=1$, so the identity matrix $I_2$ and $X$ are definitely giving two different inner products on $\Bbb R^2$ for the same input vector.


The set of symmetric matrices $S$, and the set of positive definite symmetric matrices $S^+$ could be given the subspace topology from whatever topology you choose on the set of $n\times n$ matrices. Perhaps one final thing to note is that while $S$ is a linear subspace of the $n\times n$ matrices, $S^+$ isn't.

$\endgroup$
  • $\begingroup$ Thank you for your response. Just wonder how should I proceed from here to show that the set of symmetric positive definite matrices is a sub-manifold of $\mathbb R^n$ for some $n$? $\endgroup$ – LaTeXFan Apr 30 '14 at 14:06
  • $\begingroup$ @20824 Well... I just described that it is sitting inside the $n\times n$ matrices, and that is a vector space isomorphic to $\Bbb R^{n^2}$... $\endgroup$ – rschwieb Apr 30 '14 at 14:34
1
$\begingroup$

Think of $\langle v, w \rangle = v^\top w = v^\top I w$, where $I$ is the $3 \times 3$ identity matrix. Then, a weighted inner product on $\mathbb{R}^3$ can have the form $v^\top M w$, where $M$ is a matrix. If $M$ is a $3 \times 3$ diagonal matrix, then the arising inner product is the weighted euclidean inner product on $\mathbb{R}^3$.

The set of inner products on $\mathbb{R}^3$ is a subset of the set of functions $f : \mathbb{R}^3 \times \mathbb{R}^3 \to \mathbb{R}$. If every inner product on $\mathbb{R}^3$ is of the form $v^\top M w$ for some matrix $M \in \mathbb{M}_{3,3}(\mathbb{R})$, then you can regard the set of matrices giving rise to these products as the set of inner products themselves.

I am unaware of any useful topology on this space of functions. Sorry and hope this helped!

$\endgroup$
  • $\begingroup$ My teacher said this set of inner product can be viewed as a sub-manifold of $\mathbb R^n$ for some $n$. How is that true? $\endgroup$ – LaTeXFan Apr 30 '14 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.