Dealing with a nonlinear oscillator 
Given the following differential equation
$$\ddot x + (x^2+\dot x^2-1) \dot x + x = 0$$
find the polar equations of motion in state space.

This is for my classical mechanics course, but it seems more mathematics than physics, which is why I am asking this here. I am totally lost. I thought about using a substitution $y = \dot x$, which results in
$$\dot y = (1-x^2)y-y^3$$
which doesn't seem oscillatory.  I also see that it is very similar in form to the van der Pol oscillator, but there are additional terms.  So I am just looking for a hint on how to go about this.  Thanks.
 A: Great question, interesting problem, one of my favorites, wish I could upvote the question more than once!
So the equation of interest is
$\ddot x+(x^2+\dot x^2-1)\dot x + x=0. \tag{1}$
Well, we want a polar expression for (1).  So we begin by making the two-dimensional, planar aspects of (1) obvious, since polars can't really be applied in any nice way to an equation on the line; this transformation is affected by introducing a new variable $y$, and setting $y = \dot x$, so:
$\dot x = y, \tag{2}$
then substituting $y$ from (2) into (1) we obtain
$\dot y = - (x^2 + y^2 - 1)y - x. \tag{3}$
Taken together, (2) and (3) form a planar system, often called the phase representation of the equation (1); a point in the $x$-$y$ plane now may be interpreted as a complete state of (1), since it contains all the information needed to determine the ensuing trajectory uniquely, i.e., position and velocity.  We cast (2), (3) in polar form by setting
$x = r \cos \theta, \tag{4}$
$y = r \sin \theta, \tag{5}$
from which readily follows
$\dot x = \dot r \cos \theta -r \dot \theta \sin \theta, \tag{6}$
$\dot y =\dot r \sin \theta + r\dot \theta \cos \theta. \tag{7}$
We now insert (4)-(7) into (2), (3) and, as the old song says, "boil that cabbage down", so to speak, by cooking things on the stove of algebraic manipulation:
$\dot r \cos \theta -r \dot \theta \sin \theta = r\sin \theta, \tag{8}$
$\dot r \sin \theta + r\dot \theta \cos \theta = - (r^2 \cos^2 \theta + r^2 \sin^2 \theta - 1) r \sin \theta - r \cos \theta;\tag{9}$
now, as Bob Marley sang, we Stir It Up; so, pretty baby, we re-write (8), (9) a little bit.  Notice that, in terms of the vector $(\dot r, \dot \theta)^T$ the combined
lefts of these two equations can be written in matrix-vector form, whence we arrive at
$\begin{bmatrix} \cos \theta & - r\sin \theta \\ \sin \theta & r \cos \theta \end{bmatrix} \begin{pmatrix} \dot r \\ \dot \theta \end{pmatrix} = \begin{pmatrix} r \sin \theta \\ -(r^2 - 1)r \sin \theta -r \cos \theta \end{pmatrix}.  \tag{10}$
Now as long as $r \ne 0$, the matrix on the left-hand side of (10) is nonsingular; indeed its determinant is $r$:
$\det \begin{bmatrix} \cos \theta & - r\sin \theta \\ \sin \theta & r \cos \theta \end{bmatrix} = r(\cos^2 \theta + \sin^2 \theta) = r; \tag{11}$
thus it may be simply inverted; indeed, a standard calculation yields
$\begin{bmatrix} \cos \theta & - r\sin \theta \\ \sin \theta & r \cos \theta \end{bmatrix}^{-1} = \frac{1}{r} \begin{bmatrix} r\cos \theta &  r\sin \theta \\ -\sin \theta & \cos \theta \end{bmatrix}; \tag{12}$
keep stirring, keep stirring, keep boiling:  multiply (10) by (12) and grind; we obtain
$\begin{pmatrix} \dot r \\ \dot \theta \end{pmatrix} = \begin{pmatrix} -r(r^2 - 1) \sin^2 \theta \\ -(r^2 -1)(\cos \theta)(\sin \theta) -1 \end{pmatrix}, \tag{13}$
the equation (1) in polar form may thus be written out in non-vector form as
$\dot r = -r(r^2 - 1) \sin^2 \theta, \tag{14}$
$\dot \theta = -(r^2 -1)(\cos \theta)(\sin \theta) -1 = -\frac{1}{2}(r^2 -1)\sin 2 \theta -1. \tag{15}$
There! Cabbage ready!  It is worth noting that (14) implies, as long as $\sin^2 \theta \ne 0$, $r$ increases as long as $0 < r < 1$ and decreases when $r > 1$; when $r = 1$ we have
$\dot r = 0, \tag{16}$
and
$\dot \theta = -1; \tag{17}$
the circle $r = 1$ is apparently a stable, closed, periodic orbit of period $2\pi$, rotating in the clockwise direction.  In the event $\sin^2 \theta = 0$, we have that $\theta = m\pi$ for some integer $m$; then $\sin 2\theta = 0$ as well, so that from (15) $\dot \theta = -1$; the condition $\dot r = 0$ is thus seen to be an "instantaneous" phenomenon; $r$ will continue to increase/decrease along a trajectory except at these particular values of $\theta$, with $r \to 1$.  Properties of solutions near points where $\dot \theta = 0$ may be investigated using (15); I leave such questions to my readers.  Other interesting features of the flow may be derived from examination of (15), (16); but my night job beckons, and so I must bid you all adieu.
Hope this helps.  Cheerio,
and as always,
Fiat Lux!!!
A: To complement Robert Lewis's answer, here is a Grapher plot of the vector field:

Indeed, it looks as though the unit circle is a stable limit cycle. 
Visual inspection of the plot above suggests that the origin is unstable. The Jacobian at the origin has complex eigenvalues with positive real parts. Hence, indeed, the origin is an unstable focus.
Making $\dot y = 0$, we obtain the implicitly-defined curve $x - (1-x^2-y^2) y = 0$, which is plotted below:

This algebraic curve partitions the phase plane into two regions. In one of these regions the "arrows" point upwards, while in the other region the "arrows" point downwards.
