"how do you know that $x^2 - y^3 = 5$ cannot be simplified"
For a finite system with finitely many variables, one can take derivatives of the left-hand side. Namely, hold $y$ constant, differentiate with respect to $x$ twice, and collect the like powers--this is now a little easier to do, and that's is the only merit of this method. If you don't get a zero, the polynomial is nonlinear. Similarly for $y$: hold $x$ constant, differentiate with respect to $y$ twice. Do we get a zero? If not, the polynomial is nonlinear. For the polynomial to be linear, it is necessary and sufficient that each of these second-derivative tests (for each variable involved) gives us zero.
Trying the above test on $(x+y+5)^2-(x+y)^2=35$, we see that the second derivative w.r.t. $x$ on the left gives us $0$, and the same for $y$.
Generally, though, the definition of a concept does not always give an immediate recipe for testing each object as to whether it satisfies the definition. If you've seen limits in calculus, you probably noticed that the definition of a limit gives no recipe for determining whether a limit exists, and if it does, what its value is.
The issue of "linear equations" gets a little easier to deal with once the concept of a linear transformation is introduced. Unfortunately, many courses try to teach "linear equations" before introducing linear transformations.
An equation is linear if it is (or can be) written in the form
$$Ax = b,$$
where $A$ is a linear transformation, $x$ is a sought vector in the domain of $A$, and $b$ is the given vector in the range of $A$.
This is similar to what you quoted from your text. However, it makes it easier to see that the linearity of the equation can be tested by taking two solutions, $x_{1}, x_{2}$ and seeing whether their difference is a solution of the homogeneous version
$$
Ax = \vec{0}
$$
of the original equation (think of this difference as a finite difference that approximates the first of the aforementioned derivatives). So, we are a little better off. If $A$ is given in a complicated, but explicit form, we can always consider numerical tests of linearity. Plus, this latter, coordinate-free, definition echoes the Superposition Principle, which pervades physics and engineering.
Just out of curiosity, I looked into Kurosh's Higher Algebra (a classic in its own time and a champion of clarity). It jumps into systems of linear equations from the get-go and doesn't seem too concerned with polynomials that, upon collecting the like powers, turn out linear. I haven't seen any textbook on linear algebra sweat this too much; perhaps, experience shows that testing linearity is usually not that much of difficulty, probably because linearity is so scarce.