Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I am reading a text related to SVM, and the mathematical language is giving me a little hard time.

Here training vectors xi are mapped into a higher (maybe infinite) dimensional space by the function $\theta$. SVM finds a linear separating hyperplane with the maximal margin in this higher dimensional space.

I do not understand the term "dimensional space" in this case. Drawing on a paper is 2D. We are living in 3D space. In Mathematics, when we say "higher dimensional space", what are we actually implying in Mathematics?

Another term "hyperplane" is also giving me a bit of hard time to understand. Is it simply just a 2D plane? I try to search for its definition, and most of the time, I get a term that leads to many more terms (and more confusing) Frankly, mathematical language is difficult for me.

Could somebody simplify and relate "dimension" and "hyperplane" in the text above in an easier way to understand?

Thank you very much.

share|improve this question
3  
The phrase is not "dimensional space" but "higher dimensional space", i.e. the target space that $\theta$ is mapping into is higher dimensional than the space that the $x_i live in. (The parenthetical "maybe infinite" is intended to indicate that this higher dimensional space may even be infinite dimensional.) "Hyperplane" means a subspace of dimension one less, e.g a line in a plane, a plane in space, a 3-space in 4-space, etc. If you find these concepts confusing, you may want to pick up a linear algebra text, where you can learn about such things. –  Matt E Jan 6 '11 at 7:47
add comment

3 Answers 3

up vote 8 down vote accepted

The way I read the quote, "higher (maybe infinite) dimensional space" should be read as "higher-dimensional (maybe infinite-dimensional) space" meaning a space with more dimensions than the $x_i$ had originally (mapping them into a higher-dimensional space).

A hyperplane in $n$-dimensional space is an $(n-1)$-dimensional object that can be described by $\vec{n}\cdot\vec{x}=k$ where $\vec{n}$ is a constant vector orthogonal to the hyperplane, $\vec{x}$ is a variable vector from the origin to a point on the plane, and $k$ is some scalar constant. A hyperplane in 2-space is a line; a hyperplane in 3-space is a plane. (edit: Matt E.'s comment that a hyperplane is a subspace of dimension one less than the whole space is a much nicer definition than mine.)

share|improve this answer
    
Dear Isaac, Your definition of hyperplane has the advantage of working (suitably interpreted) in infinite dimensions, whereas mine doesn't! Regards, –  Matt E Jan 6 '11 at 8:22
add comment

What that means is that SVMs in order to classify your objects need to put a boundary separating objects of class A from objects of class B, for example. This boundary will be an hyperplane. But picture a circle in 2D and let's say you want two opposite quarters to be classified as A, and the two others as B. You can't draw a line that leaves A on one side and B on another. What SVMs do to solve this is "sending" the objects in $\mathbb{R}^3$ for example and then here put a plane separating the objects (they won't be the same though, so when you "send" them you actually transform them quite much).

More generally, if you have objects in $\mathbb{R}^n$, you will just send them in $\mathbb{R}^p$, with $p > n$, through some transformation, so that you can "draw" an hyperplane leaving objects of class A on one side, and objects of class B on the other. So here's what is meant by higher dimensional vector space.

Now, an hyperplane is a really simple object. That's the generalization of straight lines in $\mathbb{R}^2$, planes in $\mathbb{R}^3$. That is, in $\mathbb{R}^n$, you just forget about one dimension -- what kind of object do you have now ? Well, an hyperplane. Formally, it's a subspace of dimension $n-1$. Why an hyperplane ? Because you see that in $E = \mathbb{R}^2$, a straight line splits $E$ into two disjoint parts. In $E = \mathbb{R}^3$, a plane splits $E$ in two disjoint parts. And the same holds for higher dimensions (though we have a lot more trouble picturing it). And SVMs will find the hyperplane with the highest margin from both sides, A and B objects.

Hope this helps.

share|improve this answer
add comment

Karl, higher dimensions are defined so as to be analogous to 2-D and 3-D. This means analytical definitions of orthogonality, distance, and so on must be preserved.

To "visualise" 4-D and up, you can start by visualisation exercises with 4-cubes or think about arrays in programming. $array[i][j][k][l] is a four-dimensional array for example. If you took all of the possible combinations of 0 and 1 in that array -- for instance [0,1,0,0], [1,1,0,0], [0,1,1,1] and so on -- you would have a data structure that's equivalent to a 4-cube or tesseract. You can generate these in R using the combn() function.

Here's another example: imagine a 4-sequence of 4 stock prices progressing in parallel, second by second. Record all the seconds for a single trading day and you have a day-long 4-D vector. That's four-dimensional data. I could have looked at 500 stock prices progressing in parallel and that would be 500-dimensional data.picture of 4 securities evolving today

The Titanic data set in R is another accessible example of 4-D data (age, sex, class, survival).

As for hyperplanes, think about this:

  • a line splits a circle
  • a plane splits a sphere
  • a hyperplane splits a ... whatever comes next.

Squiggly or blobby higher-dimensional shapes are called "manifolds", by the way.line splitting a blob in 2-D

Below are examples of functions that map data to higher dimensions.

  • $f(x,y) = (x, x, x, y, y)$. $f$ is like copying and pasting whole columns in a table or array. $f$ maps 2 dimensions to 5.
  • $g(x,y) = (x, \ x+1, \ x^2, \ x \cdot y, \ y^3 + y^5 - y^4 + 55)$. $g$ maps 2 dimensions to 5.
  • $h(a, b, c, d, e) = (a + b + c + d + e, \ a \cdot b \cdot c \cdot d \cdot e)$. $h$ maps 5 dimensions to 2.
  • Here is another function $j$ that maps 5 dimensions to 2: $j(a, b, c, d, e) = (a, e)$. $j$ is like erasing the middle columns of your data. In math books this may be called projection.
  • Here is one, $\theta$, that's used in a common SVD example: separating an inner ring from an outer ring. $\theta(x,y) = (x^2, \ \sqrt{2} \cdot x \cdot y, \ y^2)$ (that's mapping how many to how many dimensions?)
  • Here is one more function just to demonstrate that letters, numbers, or greeks can be used: $\gamma_{\eta}(a, b, c, d, e) = (0, 5)$. This one has a complicated name but it maps any 5-dimensional input to the same 2-D point, which is a "trivial" thing to do.

As you can see you just count the commas inside the parentheses to figure out how many dimensions you've got (plus one).

Hope this helps.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.