Lyapunov stability at origin with identically zero test function At the origin, determine stability of $$x' = y \\ y' = -\tan(x)$$
If we use the test function $V(x,y) = 0.5y^2 + \int_0^x tan(s)ds$, we get
$\dot{V}=x'\tan x +y'y = y\tan x -y\tan x = 0$, so the origin is definitely stable. But $\dot{V} \equiv 0$ everywhere, so LaSalle principle is tricky here. So how would we determine if it is asymptotically stable or not?
 A: Well, LaSalle principle and Lyapunov methods work great for dissipative systems, but your system is not dissipative. It has very special form 
$$ \dot{x} = \frac{\partial H}{\partial y}, $$
$$ \dot{y} = -\frac{\partial H}{\partial x}, $$
with function $H(x, y) = \frac{y^2}{2} + \int \limits_{0}^{x} \tan u\, du$ which is called Hamiltonian. Since $\dot{H}(x,y) \equiv 0$ everywhere we say that this quantity is conserved. It means that $H(x,y) = {\rm const}$ along trajectories. However, in case of 2D systems level sets of $H(x,y)$ are 1D and trajectories are also 1D, i.e. trajectories of the system are contained in level sets. The only thing that we can expect in Hamiltonian case is Lyapunov (but non-asymptotic) stability$^\ast$.
So, the only thing that you have to check is how level sets of $H(x,y)$ look like near origin. The answer (which is straightforward to check) is that level sets near origin are family of closed compact curves around equilibria. Therefore, all nearby trajectories stay on this closed curves, but never reach origin. Therefore, origin is Lyapunov stable but not asymptotically stable.
$^\ast$ Because of this special form typical equilibria of Hamiltonian systems can be only saddles or centers. Saddles are never stable and centers are Lyapunov stable, but not asymptotically stable.
A: Function 
$$V(x,y):=\frac{1}{2}y^2+\int_{0}^{x-\pi\lfloor x_0/\pi\rfloor}{\tan(s)ds}$$ is invariant since as you stated $\dot{V}=0$ and therefore if we integrate over time $[0,t]$ we have that 
$$y^2(t)+2\int_{0}^{x(t)-\pi\lfloor x_0/\pi\rfloor}{\tan(s)ds}=y_0^2+2\int_{0}^{x_0-\pi\lfloor x_0/\pi\rfloor}{\tan(s)ds}\qquad \forall t\geq 0$$ 
The r.h.s. of the above identity is a bounded constant iff $x_0\neq k\pi\pm \pi/2$. Thus, the l.h.s. is bounded for all $t\geq 0$ (note that I have avoided the issue of existence and uniqueness of the solutions but this can be treated fairly standard).
Let us define now the function $$V_0(x):=\int_{0}^{x-\pi\lfloor x_0/\pi\rfloor}{\tan(s)ds}$$
It holds true that $$V_0(x)\geq 0 \quad \forall x\in \Big(\pi\big\lfloor \frac{x_0}{\pi}\big\rfloor-\frac{\pi}{2},\pi\big\lfloor \frac{x_0}{\pi}\big\rfloor+\frac{\pi}{2}\Big)$$ and
$$V_0(x)>0 \qquad\forall x\in \Big(\pi\big\lfloor \frac{x_0}{\pi}\big\rfloor-\frac{\pi}{2},\pi\big\lfloor \frac{x_0}{\pi}\big\rfloor+\frac{\pi}{2}\Big),x\neq \pi\big\lfloor \frac{x_0}{\pi}\big\rfloor$$
with limit from the left
$$\lim_{x\rightarrow \Big(\pi\big\lfloor \frac{x_0}{\pi}\big\rfloor+\frac{\pi}{2}\Big)^{-}}V_0(x)=+\infty$$
and limit from the right
$$\lim_{x\rightarrow \Big(\pi\big\lfloor \frac{x_0}{\pi}\big\rfloor-\frac{\pi}{2}\Big)^{+}}V_0(x)=+\infty$$
The continuity of $x,y$, the nonnegativity of $V_0$, $y^2$ and the boundedness of $V$ provide us with the boundedness of $y, V_0$ and  the restriction of the trajectories within $$S:=\{(x,y)||x−π\lfloor x_0/\pi\rfloor|<π/2\}.$$
Note also that $V$ is a metric of the distance from the equilibrium $(\pi\lfloor x_0/\pi\rfloor, 0)$ that is invariant in time. This means that the trajectories do not asymptotically converge towards the equilibrium but rather remain in constant 'distance' (Lyapunov stability).
A: If you'are satisfied with asymptotic stabilty on a positive invariant set, near the origin, then you might be able to show e.g. for the set $ S = \{x,y \in \mathbb{R}: y \geq 0,  0\leq x < \pi/2\}$ with the Lyapunov function $V = y^2+\int_{0}^{x}\tan(s)\mathrm{d}s$ that $\dot V = -y\tan x$ which is negative definite on $S$.  
