You know, this is something that really annoys me about applied or non-mathematical subjects that invent superfluous terminology in order to clarify something to an audience whose background in careful mathematics is at best weak. Computer science texts are notorious for this, especially with the explosive growth of computational geometry the last decade or so. The amount of mathematical background that would be needed to make this definition precise really isn't much-basically some naive set theory and linear algebra, both of which are probably going to need to be mastered by any good computer scientist at some point anyway.
A coordinate transformation- or change of coordinates, if you must- is simply a mapping of an ordered basis to another in a vector space V where the coordinates are the scalars of the linear combinations of basis vectors that generate the space. In physics and geometry, where these transformations are so critical, the bases in question are functions that define a coordinate system on V. Therefore, the coordinate transformation is actually a linear operator from V onto V. A very nice and brief set of notes that gives examples and some more clarifying remarks can be found here.
To be honest, though, I was a bit baffled by Christian Blatter's comment above. I was always taught that a true coordinate transformation in the sense of linear algebra is always linear by definition. This extends also to manifolds, where coordinate transformations are local isomorphisms between tangent spaces.
More general, nonlinear transformations of basis functions can be allowed, of course, as parametrizing families of functions by the Implicit Function Theorem. But in this case, we are referring to functions that are not necessarily basis functions in a vector space. Physicists tend to be a little more fast and loose with this distinction then mathematicians since both kinds of transformations are important in mechanics and it's underlying geometry. The distinction is important, though, and the student should beware. Technically, a nonlinear change of coordinates is not a coordinate transformation in the mathematical sense.