8
$\begingroup$

When two finite dimensional vectors are orthogonal, i.e. perpendicular, their dot product is exactly zero, e.g. $$\mathbf{a}\cdot\mathbf{b}=a_1b_1+\cdots+a_nb_n=0.\tag{1}$$

When I studied functional analysis a long time ago, we used functions as "vectors" by way of an inner product. Consider the two functions $f(x)=\sin x$ and $g(x)=\cos x$. Functions $f$ and $g$ are said to be orthogonal over $[a,b]$ if (and only if?) $$\langle f,g\rangle=\int_a^bf(x)g(x)dx=0,\tag{2}$$ for some interval $[a,b]$. I kind of just accepted this definition, but the other day I was thinking about this...

Now, when it comes to vectors of finite dimension satisfying equation (1) the geometric interpretation of the two vectors is that they are perpendicular, e.g. imagine two $n$-dimensional lines in $\mathbb{R}^n$ and normal to each other.

Supposing (2) holds, then what does this say about $f$ and $g$ except that they are simply called orthogonal? Does (2) imply some (not necessarily geometric) relationship between functions $f$ and $g$ ?

EDIT

Didn't see these before, but related:

$\endgroup$

marked as duplicate by GEdgar, Community Nov 3 '15 at 23:00

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 3
    $\begingroup$ One could think of the function $f$ as a vector in an infinite dimensional space. The 'x-component' of $f$ is the value of $f$ at $x$, i.e, $f(x)$ where $x$ ranges in $[a,b]$. Then the integral for the inner product is akin to the summation in finite dimensional vector spaces. Then one could seek the analogy between expansion of a function as series(sum) of orthogonal functions with expansion of a vector in terms of orthogonal vectors in finite dimensional case. $\endgroup$ – vnd Nov 3 '15 at 17:36
  • 4
    $\begingroup$ The dot product is tied to the "correlation coefficient" of two sets of data. You are thus computing the correlation of continuous sets of data. $\endgroup$ – Thomas Andrews Nov 3 '15 at 17:38
  • 2
    $\begingroup$ @Thomas Andrews so one interpretation is that two orthogonal functions have zero linear relationship. $\endgroup$ – Pixel Nov 3 '15 at 17:54
  • $\begingroup$ Why do you so blithely accept orthogonality in $\mathbb R^n$? You can't exactly visualize that, either. $\endgroup$ – Jack M Nov 3 '15 at 23:09
  • 3
    $\begingroup$ I've been expecting that comment. I can mentally grasp the idea of orthogonality in higher dimensions. Visualisation is tricky of course. I think my original confusion was in accepting functions as vectors, but once i intuitively pictured these functions as infinitely many component values along the x axis i can by extension mentally accept orthogonality of function vectors too. It's all a bit mental and vague, but helps to grasp the maths. $\endgroup$ – Pixel Nov 3 '15 at 23:33
2
$\begingroup$

Let $V$ be the space of functions spanned by $f$ and $g$; that is, the space of all linear combinations $a f + b g$ (that is, the function $x \mapsto a f(x) + b g(x)$) where $a,b$ are real numbers.

Then, assuming $f$ and $g$ are linearly independent, $V$ is a two-dimensional inner product space — you really can think of this in the same fashion as any other two-dimensional inner space. Sure, $V$ is a small slice of the whole space of functions of the appropriate type, but it's the only slice that matters when $f$ and $g$ are the only functions under consideration!

In fact, you can even decompose the original space of functions into the sum of $V$ and its orthogonal complement $V^\bot$, in much the same way you can think of $\mathbb{R}^3$ as the sum of the $xy$-plane and the $z$ axis. (just that, in the analogy, you have a complicated space of functions rather than the simple $z$-axis)

e.g. if you took some orthonormal basis $u,v$ for $V$, then every function can be written as an ordered triple $h = (h_1, h_2, h_3)$ where $h_1,h_2 \in \mathbb{R}$ and $h_3 \in V^\bot$, corresponding to the function $h = h_1u + h_2v + h_3$. Then the inner product becomes

$$ \langle h, k \rangle = h_1 k_1 + h_2 k_2 + \langle h_3, k_3 \rangle$$

$\endgroup$
8
$\begingroup$

One view is to take the $L^2$ distance:

$$d(f,g)^2=\frac{1}{b-a}\int_{a}^b |f(x)-g(x)|^2 \,dx$$

if $f(x)$ and $g(x)$ are orthogonal, then what this means is that for any real $\lambda$:

$$d(f,\lambda g)\geq d(f,0)$$

That is, no multiple of $g$ is "closer" to $f$ than zero is to $f$.

This explains my comment above - there is no linear multiple of $g$ that is a "better" approximation to $f$.

Of course, this somewhat begs the question, why the $L^2$ distance?

Statistically, we can think of $d(f,0)$ as the "standard deviation" away from $0$ of the function $f$. So if $f$ is an error, then no adjustment by a multiple of $g$ diminishes that error.

Again, that begs the question - what is up with the squares and square roots in measuring standard deviation?

There is certainly something much deeper going on with squares and square roots which makes it a convenient and useful measure of distance in general.

More importantly, if $f_i$ are orthogonal, it is much easier to minimize:

$$d(g,\sum \lambda_i f_i)$$

Because the dot product gives us the value: $$d\left(g,\sum\lambda_i f_i\right)^2 = \langle g,g\rangle +\sum_i \left(\lambda_i^2\langle f_i,f_i\rangle -2\lambda_i\langle g,f_i\rangle\right)$$

Note that we can pick the $\lambda_i$ independently to minimize each term, since there is no interaction between them, and a simple bit of calculus gives us: $$\lambda_i=\frac{\langle<g,f_i\rangle}{\langle f_i,f_i\rangle}$$ which is why we prefer orthogonal functions with $\langle f_i,f_i\rangle =1$.

It should be noted that Fourier Series shows that there is a relationship between $L^2$ and a kind of infinite version of Euclidean space, $\ell^2$ where the vectors have infinite components, $\mathbf a=(a_i)_{i=0}^\infty$ and the norm is $\|\mathbf a\|=\sqrt{\sum a_i^2}$. To make this make sense, we have to include in $\ell^2$ only "vectors" where that distance makes sense. In this vector space $\ell^2$ there is a natural dot product that is "inherited" from the finite case:

$$\langle \mathbf a,\mathbf b\rangle = \sum_{i=0}^\infty a_ib_i$$

$\endgroup$
  • $\begingroup$ "no multiple of g is "closer" to f than zero is to f." $\endgroup$ – zzzzBov Nov 3 '15 at 21:13
  • $\begingroup$ Thx, will fix when home. @zzzzBov $\endgroup$ – Thomas Andrews Nov 3 '15 at 21:17
2
$\begingroup$

It is very diffuclt to speak of geometry in function spaces. We are all famiiliar with geometry in 2D spaces and 3D spaces which you meet in everyday life. One might also say he understands geometry of higher dimensions such as 4,5,etc. However trying to understand geometry of infinite dimensional space is much harder. Try to think of it the other way around. Let us look for the best approximation of $f$ by functions in the subspace spanned by $g$. You've probably seen that $(f,g)g=\int f(x)g(x)dx\cdot g(x)$ is the best approximation of $f$ in the subspaced spanned by $g$. Then saying $f$ is orthogonal to $g$ means that the zero function is the best approximation. Same holds for vectors in $\mathbb{R}^n$. I hope that helps you.

$\endgroup$
  • $\begingroup$ You still have the issue of explaining why that expression is the best approximation. In doing so, you'll find that you're implicitly working with the $L^2$ inner product (or norm). I'd honestly say it's just the most direct analogue and that there's nothing truly geometric (re: orthogonality) about it. $\endgroup$ – Cameron Williams Nov 3 '15 at 17:50
2
$\begingroup$

Take a partition of $[a,b]$ into $n$ equally-spaced subintervals. Approximate a function by assigning to each subinterval its value at the middle of that subinterval, resulting in a vector in $\mathbb{R}^n$.

To check whether two functions are orthogonal, you simply take their inner product in $\mathbb{R}^n$. That is, you multiply the functions on the subintervals and then sum the products.

When you let $n\to \infty$, the inner product becomes the standard $L^2$ inner product. So I think of orthogonality in terms of the algebraic inner product on $\Bbb{R}^n$, as saying that if we take an $\epsilon$-net, the resulting $\Bbb{R}^n$ inner product will be controlled by $\epsilon$.

$\endgroup$
2
$\begingroup$

Well, it means that the "signed area" of the region between the function T(x) = f(x).g(x) and the X axis is 0 (when viewed in the interval [a,b] of course).

This is the only geometric meaning I can think of.

$\endgroup$
  • $\begingroup$ Thanks, nice property to point out. Although, I was hoping to avoid referring back to the inner product (the integral - which is related to the area). I was thinking more of properties of $f$ and $g$ which didn't rely on the inner product. Much like perpendicularity can be viewed geometrically without referring back to the scalar product. If that makes sense. $\endgroup$ – Pixel Nov 3 '15 at 17:42
  • $\begingroup$ This feels like a smarty-pants answer, but it also is a wrong answer, in that the region is not a polygon. $\endgroup$ – Thomas Andrews Nov 3 '15 at 19:41
  • $\begingroup$ @ThomasAndrews :) Smarty-pants is that not (neither it was meant to be). It is just the only geometrical thing I could think of based on my limited knowledge. Regarding polygon - correct, it's not a polygon but ... you know what I meant, I am just not sure of the word in English. OK, I called it a figure now. $\endgroup$ – peter.petrov Nov 4 '15 at 9:28
  • $\begingroup$ @ThomasAndrews OK, you called it a region actually. So it's a region, thanks. $\endgroup$ – peter.petrov Nov 4 '15 at 9:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.