If we have three points like $A(x_1,y_1,z_1)$, $B(x_2,y_2,z_2)$ and $C(a,b,c)$. Then, $A$ and $B$ determines a line like $l$. After that, we rotate $C$ around $l$ by $\omega$ degree (anti-clockwise). How can be calculated new position of $C$ "$C'(a',b',c')$"?
There are several ways to do it, it depends on what you want to do in practice. With linear algebra, this can be achieved using a simple matrix multiplication. You can define a unit vector from $A$ to $B$ to define a rotation axis, then apply the axis-angle rotation matrix on the vector of the point $C$.
This can also be done using quaternions.
See axis-angle representation for more information.
It would be clockwise if the perspective was from a point on the line on the correct side of the perpendicular from C that intersects the line. Anticlockwise would be the perspective from the other side. As for how to do it - this is what quaternions are all about.