I think your current intuition is pretty good, let me see if I can make it a bit more precise while still keeping things conceptual. I'm not sure how you defined tangent vectors in your class but one way of doing this is as follows.
Pick a point $x \in M$. A tangent vector at $x$ is an equivalence class of curves $\gamma : (-1,1) \to M$ satisfying $\gamma(0) = x$ under the equivalence relation that they are tangent at $x$. The definition of "tangent at $x$" actually requires one to pick a chart on a neighborhood of $x$ and then transform into the statement that the gradients at $0$ are equal in $R^n$; it turns out that this is chart independent and give a real equivalence relation. So a tangent vector really does feel like an arrow in $M$ with this definition. Also interpreting a vector field as a function taking each $x \in M$ to a tangent vector at $x$ seems pretty reasonable. Recall we refer to the set of all tangent vectors at $x \in M$ as the tangent space $T_xM$ and the collection of all tangent spaces form a new manifold $TM$ called the tangent bundle (lots of grunt work goes into proving this).
Out of this nice geometric picture you can also see how to define a "differential operator" from a vector field $v$. Work pointwise in $M$. Given any differentiable function $f : M \to R$ you can operate on the function $f$ with the tangent vector in $v(x) \in T_xM$ by taking a representative curve $\gamma : (-1,1) \to M$, making the composition $f \circ \gamma : (-1,1) \to R$ and taking the derivative at zero $(f \circ \gamma)^\prime(0)$ (this gives you a number). Now one has to show that this depends only on the equivalence class of $\gamma$ (i.e. depends only on $v(x)$) and in fact defines a smooth function $M \to R$ as $x$ varies over $M$. One can work in charts to validate that the operation we've defined satisfies the product rule etc. So in this way the vector field $v$ give a differential operator. Note that all of this boils down to looking at how a single tangent vector allows us to assign a number to a function; this way of thinking of a tangent vector is usually described by calling it a derivation.
It turns out you can start just the idea of differentiation/derivations and define your tangent space from that (which is what I am guessing your class did). Doing so has certain advantages such as avoiding charts and equivalence classes. Also, if the notion of a derivation is interpreted appropriately the construction via differentiation has a certain generality because it is purely "algebraic".