What does it mean when two functions are "orthogonal", why is it important?

Question

I have often come across the concept of orthogonality and orthogonal functions e.g in fourier series the basis functions are cos and sine, and they are orthogonal. For vectors being orthogonal means that they are actually perpendicular such that their dot product is zero. However, I am not sure how sine and cosine are actually orthogonal. They are 90 out of phase, but there must be a different reason why they are considered orthogonal. What is that reason? Does being orthognal really have something to do with geometry i.e 90 degree angels?

Why do we want to have orthogonal things so often in maths? especially with transforms like fourier transform, we want to have orthogonal basis. What does that even mean? Is there something magical about things being orthogonal?

Answer 1

The concept of orthogonality with regards to functions is like a more general way of talking about orthogonality with regards to vectors. Orthogonal vectors are geometrically perpendicular because their dot product is equal to zero. When you take the dot product of two vectors you multiply their entries and add them together; but if you wanted to take the "dot" or inner product of two functions, you would treat them as though they were vectors with infinitely many entries and taking the dot product would become multiplying the functions together and then integrating over some interval. It turns out that for the inner product (for arbitrary real number L) $$\langle f,g\rangle = \frac{1}{L}\int_{-L}^Lf(x)g(x)dx$$ the functions $\sin(\frac{n\pi x}{L})$ and $\cos(\frac{n\pi x}{L})$ with natural numbers n form an orthogonal basis. That is $\langle \sin(\frac{n\pi x}{L}),\sin(\frac{m\pi x}{L})\rangle = 0$ if $m \neq n$ and equals $1$ otherwise (the same goes for Cosine). So that when you express a function with a Fourier series you are actually performing the Gram-Schimdt process, by projecting a function onto a basis of Sine and Cosine functions. I hope this answers your question!

Answer 2

Vectors are orthogonal not if they have a $90$ degree angle between them; this is just a special case. Actual orthogonality is defined with respect to an inner product. It is just the case that for the standard inner product on $\mathbb{R}^3$, if vectors are orthogonal, they have a $90$ angle between them. We can define lots of inner products when we talk about orthogonality if the inner product is zero. In the case of Fourier series the inner product is:

$$ \langle \, f ,g\rangle = \int_{-\pi}^{\pi} f(x) g(x)^* dx$$

and indeed $\langle \sin ,\cos\rangle = \int_{-\pi}^{\pi} \sin(x) \cos(x) dx = 0 $ as the integrand is odd.

And yes there is something special about things being orthogonal, in the case of the Fourier series we have an orthogonal basis $e_0(x), \dots, e_n(x),\dots$ of all $2\pi$ periodic functions. Given any function $f$ if we want to write $f$ in this basis we can compute the coefficients of the basis elements simply by calculating the inner product. Since:

$$ f(x) = \sum_{k= 0}^{\infty} a_k e_k(x)$$

$$ \langle \, f ,e_i\rangle = \int_{-\pi}^{\pi} f(x) e_i(x)^* dx = \int_{-\pi}^{\pi} \sum_{k= 1}^{\infty} a_k e_k(x) e_i(x)^* dx $$

$$ \sum_{k= 0}^{\infty} a_k \int_{-\pi}^{\pi} e_k(x) e_i(x)^* dx$$

And now magically by the orthogonality:

$$ = \sum_{k= 1}^{\infty} a_k \delta_{i,k} = a_i$$

So we can write any function directly in the orthogonal basis:

$$ f(x) = \sum_{k= 0}^{\infty} \langle \, f ,e_k\rangle e_k(x)$$

Answer 3

Orthogonality, as you seem to be aware, comes originally from geometry. We talk about two vectors (by which I mean directed line segments) being orthogonal when they form a right angle with each other. When you have orthogonal vectors, you can apply things like Pythagoras's Theorem, which is quite a neat theorem when you think about it, and should hint at some of the power of orthogonality.

The dot product allows us to talk about orthogonality a little more algebraically. Rather than considering directed line segments, we can consider elements of $\mathbb{R}^n$ instead. Orthogonality translates into the dot product equaling zero.

Now, the orthogonality you see when studying Fourier series is a different type again. There is a very common, widely-used concept of a vector space, which is an abstract set with some operations on it, that satisfies something like $9$ axioms, which ensures it works a lot like $\mathbb{R}^n$ in many respects. We can do things like add the "vectors" and scale the "vectors" by some constant, and it all behaves very naturally. The set of real-valued functions on any given set is an example of a vector space. It means we can treat functions much like vectors.

Now, if we can treat functions like vectors, perhaps we can also do some geometry with the, and define an equivalent concept of a dot product? As it turns out, on certain vector spaces of functions, we can define an equivalent notion to a dot product, where we can "multiply" two "vectors" (read: functions), to give back a scalar (a real number). Such a product is called an "inner product", and it too is defined by a handful of axioms, to make sure it behaves how we'd expect. We define two "vectors" to be orthogonal if their inner product is equal to $0$.

When studying Fourier series, you're specifically looking at the space of square-(Lebesgue)-integrable $L^2{[-\pi, \pi]}$, which has an inner product, $$\langle f, g \rangle := \int_{-\pi}^{\pi} f(x)g(x) \mathrm{d}x.$$ To say functions $f$ and $g$ are orthogonal means to say the above integral is $0$. Fourier series are just a series to express functions in $L^2{[-\pi, \pi]}$ as an infinite sum of orthogonal functions.

Now, we use orthogonality of functions because it actually produces really nice results. Fourier series are a very efficient way of approximating functions, and very easy to work with in terms of calculation. Things like Pythagoras's theorem still hold, and turn out to be quite useful! If you want to know more, I suggest studying Fourier series and/or Hilbert Spaces.

Answer 4

You're correct, "orthogonal" does indeed require a definition. The idea is that you need some idea of an inner product.

An inner product, on a vector space, $V$ over a field of scalars, $F$, is a function $\langle \cdot , \cdot \rangle \colon V \times V \to F$ satisfying a few properties (easily found in any linear algebra book or on Wikipedia). For example the dot product on $\Bbb{R}^n$ is an inner product.

We call two vectors, $v_1,v_2$ orthogonal if $\langle v_1, v_2 \rangle=0$.

For example $(1,0,0) \cdot (0,1,0)=0+0+0=0$ so the two vectors are orthogonal.

So if we have a vector space of functions, a function space. For example, $L^2([-\pi,\pi])$, the square integrable, complex valued, functions on $[-\pi,\pi]$, we can define an inner product as:

$$\langle f, g \rangle = \frac{1}{\pi}\int_{-\pi}^{\pi} f^{\ast}(x)g(x) dx $$

Two functions are orthogonal if $\frac{1}{2\pi}\int_{-\pi}^{\pi} f^{\ast}(x)g(x) dx=0$. For example $\sin(x),\cos(x)$ are orthogonal.

Why do we care? Well, Fourier series for one. For Fourier series, you expand a function on $L^2([-\pi,\pi])$ as a sum of cosines and sines. How do you calculate the coefficients?

Note that $\langle \sin(nx),\sin(mx)\rangle = \delta_{n,m}$. It is $1$ if $n=m$ and $0$ else. Same with cosine, and mixed $\sin$ and $\cos$ are $0$. This forms an orthonormal basis. Meaning all the basis vectors are orthogonal, and the inner product of any basis vector with itself is $1$. This gives us a way of computing Fourier series.

Basically we assume $f(x)=\sum a_n \cos(nx)+b_n \sin(nx)$. Then note:

$$\langle \sin(mx),f(x) \rangle = \langle \sin(mx),\sum a_n \cos(nx)+b_n \sin(nx) \rangle = b_m$$

So we can figure out the entire series.

Answer 5

When a function basis is orthogonal, the representation is much smaller (say, $N$ coefficients instead of $N^N$) and unique.

Suppose you have a function $f(x)$ and you want to approximate it using 3 basis functions - $g_1(x)$, $g_2(x)$, $g_3(x)$.

You try to approximate $f(x)$ using those functions: $$f(x) = c_1*g_1(x) + c_2*g_2(x) + c_3*g_3(x) + c_{1,2}*g_1(x)*g_2(x) + c_{2,3}*g_2(x)*g_3(x) + c_{3,1}*g_3(x)*g_1(x) + c_{1,2,3}*g_1(x)*g_2(x)*g_3(x)$$

As you see, there are quite a lot of coefficients (growing exponentially as the number of function grows). Also the expansion is not unique.

If the functions were orthogonal, then all non-single-function coefficients would be zero:

$$f(x) = c_1*g_1(x) + c_2*g_2(x) + c_3*g_3(x)$$

The set of the coefficients would also be unique for each function.

Answer 6

simply in quantum mechanics the orthogonal functions equal to zero means that no overlap between them. each function represents the energy state and for non degeneracy states the overlap between states is not found. in vector form means that no projection of each vector on other vector like the unit vector i, j, k are the basis vectors.

Answer 7

Here is a way to consider it, for what it's worth. Plot f and g on separate sheets of paper. For convenience, put the sheet with the graph of f horizontally and the sheet with the graph of g vertically (at a 90 degree angle) to the other sheet. Take a value of x, and for simplicity, assume that f(x) is positive. So, you can point your right thumb from the x axis out to f(x). If g(x) is also positive, you can point your right index finger up to it. In such a case, the cross product of f(x) and g(x) points in the direction of the third finger of your right hand towards you. If f(x) is positive but g(x) is negative, then the third finger of your right hand points away from you. Or, in other words, f(x), g(x), and f(x) cross g(x) form a left-handed system at that x. Now integrate over the region. If f, g, and f cross g "on average" (bearing in mind that the magnitude of f(x) cross g(x) is f(x) times g(x)) are left handed just as much as they are right handed, then f and g are orthogonal. I hope this makes sense. You might try visualizing it with sine and cosine for a specific example.