You seem to have some confusion about the definition of a vector field. There are two equivalent ways to define vector fields on a manifold $M$.
A vector field is a section of the tangent bundle $TM$. i.e. a smooth map $X\colon M \to TM$ with the property that $pr\circ X = id_M$, where $pr\colon TM \to M$ is projection. (Note, this is not precisely the same as the comment you write above or what is written in your question and I believe it is the source of your confusion).
A vector field is a linear map $X\colon C^\infty(M) \to C^\infty(M)$ with the property that it is a derivation, i.e. $X(fg) = fX(g) + X(f)g$ for all $f,g \in C^\infty(M)$.
To answer your question of why for two vector fields $X$ and $Y$, $XY$ is not a vector field, you should check one of the two definitions. The comment by @Hagen von Eitzen hints to use the second definition (which should be the easier definition to use for this question).
In particular, the product of vector fields can be defined two ways as well, depending on which definition of vector fields you are using. What you wrote in your post, that $XY(f) = X(Y(f))$ makes sense using definition 2: you are composing two linear maps,
$$ C^\infty(M) \rightarrow^Y C^\infty(M) \rightarrow^X C^\infty(M).$$
$$ f \, \mapsto\, Y(f) \, \mapsto \,X(Y(f))$$
Now, to see if this linear map fits definition 2 of a vector field you need to check that...
If you want to translate to definition 1, you have to do a little bit of work.