stability and asymptotic stability: unstable but asymptotically convergent solution of nonlinear system Consider nonlinear systems of the form $X(t)'=F(X(t))$, where $F$ is smooth (assume $C^\infty$). Is it possible to construct such a system (preferably planar system) so that $X_0$ is an unstable equilibrium, but all nearby solution curves tend to $X_0$ as $t \to \infty$? If so, how? A conceptual construction is enough. What will the phase portrait look like?
 A: 
Definition (Verhulst 1996)
The equilibrium solution $X_c$ is called asymptotically stable if there exists a $\delta(t_0)$ such that
\begin{equation}
||X_0 - X_c|| \le \delta(t_0) \implies \lim_{t\rightarrow\infty} ||X(t;t_0,X_0) - X_c|| =0
\end{equation}

So no, you're essentially asking can a system be both asymptotically stable and unstable which is a contradiction.
EDIT:
Glendinning's text refers to this definition as quasi-asymptotic stability to differentiate it from 'direct' asymptotic stability (solutions are both quasi-asymptotically stable and Lyupanov stable).
As an example he presents this equation:
\begin{equation}
\dot{r} = r(1-r^2) \qquad \dot{\theta} = 2\sin^2\left(\frac{1}{2}\theta\right),
\end{equation}
which has an unstable critical point at $(0,0)$ a quasi-asymptotically stable critical point at $(1,0)$.
Points are attracted to the invariant circle at $r=1$. On this curve, the flow is semi-stable. If $\pi>\theta >0$ then the flow wraps around until it passes $\theta = \pi$. If $ -\pi> \theta>0$ the trajectory falls into the critical point at $(1,0)$. This is an example of a homoclinic orbit.
In the example provided, the critical point at $(1,0)$ is a saddle. Which, if all we considered was the linearised system, is unstable. Indeed it is Lyapunov unstable as there is a trajectory that leaves any $\epsilon$-ball about $(1,0)$.
However, in nonlinear systems the linearisation is often not sufficient,
especially with saddles, which are often the cause of a whole bunch of interesting behavior that aren't represented in the linearisation, like homoclinic and heteroclinic connections.
I think the confusion comes from how you/they define instability.
The texts I refer to define a critical point as unstable if it is not Lyapunov stable or 'Quasi-Asympotically' stable. However if all you go off is the linearisation then i'd imagine the example is what you're after.
A: 
Definition of stability:
Suppose $X^∗$ ∈ $\mathbb{R}^n$ is an equilibrium point for the differential equation
$$X′ = F(X).$$
Then $X^∗$ is a stable equilibrium if for every neighborhood $\mathcal{O}$ of $X^∗$ in $\mathbb{R}^n$ there is a neighborhood $\mathcal{O}_1$ of $X^∗$ in $\mathcal{O}$ such that every solution $X(t)$ with $X(0) = X_0$ in $\mathcal{O}_1$ is defined and remains in $\mathcal{O}$ for all $t > 0$.

and

Definition of asymptotic stability:
If $\mathcal{O}_1$ can be chosen so that, in addition to the properties for stability, we have $\lim_{t \to \infty} X(t) = X^∗$, then we say that $X^∗$ is asymptotically stable.

The exact problem:

In the definition of an asymptotically stable equilibrium point, we required that the equilibrium point also be stable. This requirement is not vacuous. Give an example of a phase portrait (a sketch is sufficient) that has an equilibrium point toward which all nearby solution curves (eventually) tend, but which is not stable.


The $C^1$ example I found with the aid of Mathematica (I guess $C^\infty$ is not possible):
StreamPlot[{x^2 + (-(x^2 + 1/16) (y - 1/2)^3) Boole[y >= 1/2], -y + (y - 1)^2 y Boole[y >= 1] - 500 (y - 2)^4 Boole[y >= 2] + (x + 1)^3 Boole[x <= -1] + (x - 1)^3 Boole[x >= 1]}, {x, -4, 4}, {y, -4, 4}]

The phase portrait is here (I don't have privilege to post image; you can always run Mathematica with the above code to get the phase portrait). The example is unprecedentedly ugly, but at least it works.
This is inspired by the system
$$x'=x^2,$$
$$y'=-y.$$
and in a small neighborhood of 0 (actually $[-1,1] \times [-\infty, 1/2]$), the system is exactly the above simple simple. All solutions nearby 0 goes back to 0, since nearby solutions in the left half plane naturally converge to 0, and nearby solutions in the right half plane first converge to the $x$-axis, then wind back to the left half plane.
