Understand the need of affine connection I can't understand why textbook says that directional derivative cannot be defined for general manifold, and a separate affine connection is needed.
Assume there are two vector fields $X$ and $Y$. In particular, you have a tangent vector at point $p$ called $X_p$. To find the derivative of $Y$ at $p$ in the direction of $X_p$, why can't you just operate $X_p = a^i\partial_i$ on $Y_p = b^j \partial_j$ component-wise?
By component-wise, I meant differentiate the component smooth function $b^j$ with respect to $X_p$. It seems that a smooth real valued function can still be differentiated by a vector field without connection?
 A: The problem is that the vector field you get from componentwise directional derivative depends on choice of the coordinate system.
A: What you're describing is the contraction of exterior derivative of $Y$ by $X$, $\textrm{d}Y(X)$. This can be viewed as the trivial affine connection, as the connection matrix is $0$. In $\mathbb{R}^n$ this coincides with Levi-Civita connection of the Euclidean metric. 
In fact, one can always pick coordinates at a point such that a connection matrix vanishes at that point (Riemann Normal coordinates in the case of the Levi-Civita connection).
Alternatively, if one were to naively differentiate 'Y with respect to X' like any function, one has $$X (Y)= X^i{\partial_i}(Y^j{\partial_j}) = X^i ({\partial_i}(Y^j)\partial_j +X^iY^j{\partial_i}({\partial_j}).$$ The issue here is that this is no longer a first order operation, as we are dealing with the second derivative of coordinates.
This leads to the definition another type of directional derivative, the Lie derivative $\mathcal{L}_X$.
The intuition here is that $\mathcal{L}_X$ shows how the vector field $Y$ evolves over the integral curves of $X$, but one can show that $\mathcal{L}_X(Y)=[X,Y]$.
