Dynamical systems is an enormous area of research, so it's impossible to give a comprehensive answer, but I can try to guess at the sorts of phenomena that your physics book was describing. You'll likely get an even better answer if you post some of the vague comments that particularly confused you.
The basic problem in smooth dynamical systems (at least, the problem most relevant to physics) is to understand the trajectories determined by a possibly nonlinear system of ordinary differential equations. The first result is a local one: the Hadamard-Perron theorem relates the dynamics near a hyperbolic fixed point to the dynamics of the associated linearized system. More precisely, if $\textbf{x}'(t) = \textbf{F}(\textbf{x})$ is a nonlinear vector valued differential equation and $\textbf{x}_0$ is a point with the property that $\textbf{F}(\textbf{x}_0) = 0$ and the matrix $A = d\textbf{F}(\textbf{x}_0)$ has only nonzero eigenvalues then the phase portrait of the original dynamical system in a small neighborhood of $\textbf{x}_0$ is diffeomorphic to the phase portrait of $\textbf{x}'(t) = A \textbf{x}$ in a small neighborhood of $\textbf{0}$. In particular, the phase portrait has a nonlinear coordinate system given by an unstable manifold (corresponding to positive eigenvalues) and a stable manifold (corresponding to the negative eigenvalues.
The behavior along the stable manifold is very simple: everything flows toward the fixed point. So the challenge is to understand the global behavior unstable trajectories. There are a variety of theorems (many of which don't seem to have names) which assert that in favorable circumstances the flow of a dynamical system is ergodic, meaning informally that the dynamics "mixes up" the phase space quite a lot. For example, the geodesic flow on a negatively curved manifold is ergodic.
There are a variety of very beautiful theorems about ergodic dynamical systems; the centerpiece is Birkhoff's ergodic theorem which asserts that the amount of time that a dynamical system spends in a piece of the phase space is asymptotically equal to the volume of that piece. In other words, ergodic dynamical systems explore phase space uniformly. The ergodic theorem has many applications in geometry, number theory, and physics.
So that's a brief rundown of hyperbolic dynamics. Many other results in dynamics are answers to questions that arise naturally from the careful study of hyperbolic dynamics. What is the relationship between different fixed points for the same system? What happens near non-hyperbolic fixed points? When is the flow of a dynamical system ergodic? When is a dynamical system equivalent (topologically conjugate) to a hyperbolic system? And so on... Of course, as I warned at the beginning the subject is huge; many important theorems emerged from a detailed study of specific examples; "period 3 implies chaos" and Feigenbaum's phenomenon are examples of this sort.