For example, I have set of points in 3D. Points lie on straight line. Transformed set of points lies on straight line too. How to check if transformation is affine?
According to the fundamental theorem of affine geometry, it suffices to check that the transformation is a bijective collineation:
The proof can be found in Berger's Geometry 1 (Springer, 1987, pp. 52-56).
an affine transformation between two vector spaces $$F:X\rightarrow Y$$ (one might define it more general) is defined as $$y = F(x) = Ax + y_0$$
where $A$ is a constant map (might be represented as matrix) and $y_0\in Y$ is a constant element.
So, to check if a transformation is affine you might try to proof that such $A$ and $y_0$ exist.