Why were Lie algebras called infinitesimal groups? Why were Lie algebras called infinitesimal groups in the past? And why did mathematicians begin to avoid calling them infinitesimal groups and switch to calling them Lie algebras?
 A: I don't know anything about the actual history, but you might want to look at this blog post by Terry Tao. Terry defines a "local group" to be something that looks like a neighborhood of the identity in a topological group and a "local Lie group" to be something that looks like a neighborhood of the identity in a Lie group. 
He then defines a "group germ" to be an equivalence class of local groups, where the equivalence relation should be thought of as "become the same under passing to a sufficiently small open neighborhood of the identity". Thus, the group germ remembers all the information which can be seen on an arbitrarily small neighborhood of the identity. I think this is a very good modern rigorous analogue of the notion of an infinitesimal group. Terry then proves Lie's Third Theorem (Theorem 2 in his notes): Germs of local lie groups are in bijection with Lie algebras.
If you prefer algebra to analysis, the corresponding idea is a formal group. Again, it is a true but nontrivial theorem that formal groups in characteristic zero are equivalent to Lie algebra.
A: Thomas Hawkins's Emergence of the Theory of Lie Groups: An Essay in the History of Mathematics 1869–1926 (2000) answers your question.
