It seems that you are mixing a few things up. A smooth manifold $M$ locally "looks like" an Euclidean space $\mathbb{R}^n$ in the sense that every point $p \in M$ has a neighborhood that is diffeomorphic to an open subset of $\mathbb{R}^n$. This has nothing to do with connections.
Instead of using the terminology "affine connection", let me use the term "linear connection" (which is often a synonym and) because it is much easier to see what is "linear" about a connection while it is harder to explain what is "affine" about it.
An linear connections $\nabla$ on $M$ is an extra piece of data which allows you to identify tangent spaces at different points along a curve in a linear way. The mechanism which identifies the tangent spaces along a curve is called parallel transport.
This generalizes the fact that on a vector space $V$, you can identify all the different tangent spaces $T_p V$ with $V$ itself (by "parallel translating" $T_p V$ back to the origin) and in particular identify $T_p V$ with $T_q V$ for different $p,q$. In a vector space, this identification can be done "globally" without any choice while on a manifold, such an identification is possible only given an extra piece of data (a connection) and even then it can be done only along curves. This has nothing to do with "being charted by Cartesian coordinates" because all manifolds can be charted by Cartesian coordinates by definition.
A non-linear (or non-affine) connection on $M$ is an extra piece of data which allows you to identify different tangent spaces at different points along short enough curves in a not-neccesarily-linear way. This means that the parallel transport induced by the connection is not necessarily linear.
