Cosine of angle between two vectors G. E. Hay’s Vector and Tensor Analysis, Chapter 1, Section 7:

Let us consider two vectors ${\bf a}$ and ${\bf b}$ with magnitudes $a$ and $b$, respectively. Let $\alpha$ be the smallest nonnegative angle between ${\bf a}$ and ${\bf b}$. Then $0^\circ\leq\alpha\leq 180^\circ$. The scalar $ab\cos\alpha$ arises quite frequently, and hence it is convenient to give it a name. It is called the scalar product of ${\bf a}$ and ${\bf b}$. It is also denoted by the symbols ${\bf a}\cdot{\bf b}$, and hence we have $${\bf a}\cdot{\bf b}=ab\cos\alpha.$$ The scalar product is sometimes referred to as the dot product.
If the components of ${\bf a}$ and ${\bf b}$ are denoted by $a_1$, $a_2$, $a_3$, $b_1$, $b_2$ and $b_3$ in the usual manner, the direction cosines of the directions of ${\bf a}$ and ${\bf b}$ are respectively $$\frac{a_1}{a},\frac{a_2}{a},\frac{a_3}{a};\frac{b_1}{b},\frac{b_2}{b},\frac{b_3}{b}.$$ By a formula of analytic geometry, we then have $$\cos\alpha=\frac{a_1}{a}\frac{b_1}{b}+\frac{a_2}{a}\frac{b_2}{b}+\frac{a_3}{a}\frac{b_3}{b}.$$

What is this formula of analytic geometry?
 A: Start with the vector formula for the dot product:
$\vec{a} \cdot \vec{s} = a_1 b_1 + a_2 b_2 + a_3 b_3$
Now draw any triangle you want and and choose a corner. There are two lines coming out of that corner. Consider them to be the vectors $\vec{a}$ and $\vec{b}$, both pointing outward from that point. Note that these are arbitrary because the triangle is arbitrary. Then, the law of cosines gives you the length of the remaining side of the triangle (which is $|\vec{a} - \vec{b}|$):
$(1) \hspace{2in} |\vec{a} - \vec{b}|^2 = a^2 + b^2 - 2 a b \cos \alpha $.
We have that 
$(2) \hspace{1in} |\vec{a} - \vec{b}|^2=(\vec{a} - \vec{b})\cdot (\vec{a} - \vec{b})=(a_1 - b_1)^2+(a_2 - b_2)^2+(a_3 - b_3)^2$
$(3) \hspace{2in} a^2=\vec{a} \cdot \vec{a} = a_1^2 + a_2^2 + a_3^2$
$(4) \hspace{2in} b^2=\vec{b} \cdot \vec{b} = b_1^2 + b_2^2 + b_3^2$.
Using (2)-(4) to replace the LHS and RHS of (1), then expanding all the terms in the new equation and cancelling stuff out from both sides, you will find that
$(5)\hspace{2in}a_1 b_1 + a_2 b_2 + a_3 b_3 = ab\cos\alpha$, 
thus proving the equivalence of the definitons of the dot product. The formula you seek is just the law of cosines, and the final equation in your question is obtained by dividing both sides of (5) by $ab.$ The law of cosines is proved in a readable fashion here: http://pages.pacificcoast.net/~cazelais/173/law-sines-cosines.pdf
