Standard matrix of a rotation on a vector in $\mathbb{R}^3$? $\begin{pmatrix}\cos \alpha & -\sin \alpha \\ \sin \alpha & \cos \alpha\end{pmatrix}$ defines a rotation of any angle on a vector in $\mathbb{R}^2$, but how does it work in $\mathbb{R}^3$ ? (I'd imagine this idea isn't very useful in dimensions other than 2 and 3) 
 A: There is another very efficient answer : by using matrix exponentials.
This is best understood by analogy with the fact that, identifying $\mathbb{R}^2$ with $\mathbb{C}$, your transformation :
$$\begin{pmatrix}x'\\y'\end{pmatrix}=\begin{pmatrix}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta\end{pmatrix}\begin{pmatrix}x\\y\end{pmatrix}$$
becomes, with complex notations : 
$$x'+iy'=\cos(\theta)x-\sin(\theta)y + i(\sin(\theta)x+\cos(\theta)y)$$
which can be simplified into 
$$x'+iy'=(\cos(\theta)+i\sin(\theta))(x+iy)$$
Said otherwise : $$z'=e^{i\theta}z. \tag{1}$$
It happens that expression (1) has a nice generalization in $\mathbb{R^3}$ under an analogous form $X'=\exp(...)X$ as follows:

The matrix of the 3D rotation with angle $\theta$ around (any) unit vector $N=[a,b,c]$ is given by :

$$\exp(\theta [N]_{\times}),\tag{2}$$ 
with the skew-symmetric matrix :
$$ [N]_{\times} := \begin{pmatrix}0&-c&b\\c&0&-a\\-b&a&0\end{pmatrix} \tag{3}.$$
Most languages have a built-in implementation of the exponential of a matrix. Let us take an example with Matlab : rotation with angle $2\pi/3$ around the axis defined by the nonunitary vector $U=(1,1,1)$ is obtained in this way :
U=[1,1,1];
U=U/norm(U);
a=U(1);b=U(2);c=U(3);
V=[0,-c,b;
   c,0,-a;
   -b,a,0];
R=expm((2*pi/3)*V); % (expm for exp-matricial)
% giving the result :
% 0 0 1
% 1 0 0
% 0 1 0

as can be awaited (canonical axes $e_1 \to e_2 \to e_3 (\to e_1)$).
Let us give some precision about  :


*

*The exponential of a matrix $M$, which is defined by the same series expansion as the ordinary exponential : 


$$\exp(M)=I+M+\frac12M^2+\cdots+\frac{1}{n!}M^n+\cdots \tag{4}$$


*

*The matrix $[N]_{\times}$ defined in (3) which will no longer look mysterious when it is associated with the cross product under the following form :


$$Z=[N]_{\times}Y = N \times Y$$ 
as can be verified with the following explicit computation :
$$\begin{pmatrix}u\\v\\w\end{pmatrix}=\begin{pmatrix}\ \ 0&-c& \ \ b\\ \ \ c&0&-a\\-b& \ \ a&0\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}bz-cy\\cx-az\\ay-bx\end{pmatrix}$$
A connection with Rodrigues formula (given by @Artic tern) is straightforward by decomposing even and odd terms in (3), then taking into account the fact that $([N]_{\times})^3=-[N]_{\times}$: see for example : https://en.wikipedia.org/wiki/Axis%E2%80%93angle_representation . See also an answer of mine :{https://math.stackexchange.com/q/1917093}.
Final remark : this expression of a 3D rotation is made for computers, of course. But nowadays very few people compute 3D rotations by hand... 
A: Let $\{\mathbf{u},\mathbf{v},\mathbf{w}\}$ be an oriented orthonormal basis and let $R$ be the rotation around $\mathbf{w}$ by $\theta$.
Since the cross product $\mathbf{w}\times\mathbf{x}$ has the effect of projecting $\mathbf{x}$ onto the $\mathbf{uv}$-plane and rotating by a right angle, the combination $(\cos\theta)\mathbf{x}+(\sin\theta)\mathbf{w}\times\mathbf{x}$ is almost $R(\mathbf{x})$, but with the $\mathbf{w}$-component of $\mathbf{x}$ scaled by $\cos\theta$; to compensate, add in $\mathbf{w}$ scaled by $(1-\cos\theta)$ times the $\mathbf{w}$-component of $\mathbf{x}$.
This yields the Rodrigues' rotation formula
$$ R(\mathbf{x})=(\cos\theta)\mathbf{x}+(\sin\theta)(\mathbf{w}\times\mathbf{x})+(1-\cos\theta)(\mathbf{w}\cdot\mathbf{x})\mathbf{w}. $$
Feel free to write out an explicit formula from this.
Because the $2$-sphere $S^2$ parametrizes the possible "axes" $\mathbf{w}$ and $\theta$ fills in the rest of the rotation $R$, the group of all $3\times3$ rotation matrices is $(2+1)=3$-dimensional.

An orthogonal matrix is one which satisfies any of the following:


*

*$\|A\mathbf{x}\|=\|\mathbf{x}\|$ for all vectors $\mathbf{x}$ (i.e. preserves lengths)

*$(A\mathbf{x})\cdot(A\mathbf{y})=\mathbf{x}\cdot\mathbf{y}$ for all vectors $\mathbf{x},\mathbf{y}$ (preserves lengths and angles)

*$A^TA$ is the identity matrix $I$

*The columns of $A$ are orthonormal vectors

*The rows of $A$ are orthonormal vectors


The orthogonal matrices with determinant $-1$ reverse the orientation of space, such as reflections or the inversion-about-the-origin matrix $-I$. The orthogonal matrices with determinant $+1$ preserve the orientation of space and are called special orthogonal - these are precisely the $3\times 3$ rotation matrices.
So to form any rotation matrix $R$, you can pick the first column vector to be any unit vector $\mathbf{u}$ (essentially a choice of where you want $R$ to send $\mathbf{e}_1$), in the second column you can pick any unit vector $\mathbf{v}$ orthogonal to the first vector $\mathbf{v}$ (a choice of where $R$ sends $\mathbf{e}_2$), and finally $\mathbf{w}=\mathbf{u}\times\mathbf{v}$ is the unique unit vector which makes $\{\mathbf{u},\mathbf{v},\mathbf{w}\}$ a correctly-oriented orthonormal basis (and $\mathbf{w}$ is a choice of where to send $\mathbf{e}_3$).
Since $\mathbf{u}$ is chosen from $S^2$ and $\mathbf{v}$ is chosen from the circle's worth of perpendicular unit vectors, the $3\times 3$ rotation group we can again conclude is $(2+1)=3$-dimensional.
(Note the vectors $\mathbf{u},\mathbf{v},\mathbf{w}$ represent different things in these two descriptions.)
