Calculating cosine of dihedral angle 
Let $O,A,B,C$ be points in space such that $\angle AOB=60^{\circ},\angle BOC=90^{\circ},\angle COA=120^{\circ}$ Let $\theta$ be the acute angle between the planes $AOB$ and $AOC$. Find $\cos\theta.$

Here is what I did:
I let $AO=OB=BA=1, OC=1$. Then we can compute $BC=\sqrt{2},AC=\sqrt{3}$. From here, I let $A=(0,0,0),B=(1,0,0),C=(\frac{1}{2},\frac{\sqrt{3}}{2},0),C=(1,0,\sqrt{2})$ (I found the coordinates of $C$ by noticing $\angle ABC=90^{\circ}$ since $AB^2+BC^2=AC^2$). Then I found the equation of $AOC$ to be $-\sqrt{6}x+\sqrt{2}y+\sqrt{3}z=0$ and the equation of $AOB$ to be $z=0$, so using the formula here, I got $\cos\theta=\frac{\sqrt{3}}{11}$, but the answer is $\frac{1}{3}$. Where did I mess up?
Also a solution without using coordinates would be very nice to see!
 A: Your starting was right, so we have
$$
O = (0,0,0)\quad A = (1,0,0)\quad B = {1 \over 2}\left( {1,\sqrt 3 ,0} \right)
$$
for point C instead, we shall have
$$
\left\{ \matrix{
  \mathop {OB}\limits^ \to  \, \cdot \;\mathop {OC}\limits^ \to   = \cos \left( {90^\circ } \right) = 0 \hfill \cr 
  \mathop {OA}\limits^ \to  \, \cdot \;\mathop {OC}\limits^ \to   = \cos \left( {120^\circ } \right) =  - 1/2 \hfill \cr}  \right.
$$
and putting $\mathop {OC}\limits^ \to   = \left( {x,y,z} \right)$
$$
\left\{ \matrix{
  x + \sqrt 3 y = 0 \hfill \cr 
  x =  - 1/2 \hfill \cr}  \right.\quad  \to \quad \left\{ \matrix{
  x =  - 1/2 \hfill \cr 
  y = {1 \over {2\sqrt 3 }} \hfill \cr}  \right.
$$
Now $z$ comes to be "free", but if you want $\left| {\mathop {OC}\limits^ \to  } \right| = 1$ then $z = {{\sqrt 2 } \over {\sqrt 3 }}$ , so
$$
\mathop {OC}\limits^ \to   = \left( { - {1 \over 2},{1 \over {2\sqrt 3 }},{{\sqrt 2 } \over {\sqrt 3 }}} \right)
$$
Once you get three points, i.e. two vectors, lying on a plane, the cross-product of the two vectors will give a vector normal to the plane. So
$$
{\bf n} = \mathop {OA}\limits^ \to  \, \times \,\mathop {OB}\limits^ \to   = \left( {0,0,{{\sqrt 3 } \over 2}} \right)\quad {\bf m} = \mathop {OA}\limits^ \to  \; \times \;\mathop {OC}\limits^ \to   = \left( {0, - {{\sqrt 2 } \over {\sqrt 3 }}, - {1 \over {2\sqrt 3 }}} \right)
$$
and the dot-product of the normal unit vectors to two planes will give you the cosine of their dihedral angle (apart the sign), thus
$$
{{\bf n} \over {\left| {\bf n} \right|}} \cdot {{\bf m} \over {\left| {\bf m} \right|}} =  \pm \cos \theta \quad  \to \quad \left| {\left( { - {1 \over 4}} \right)\left( {{{\sqrt 3 } \over 2}\sqrt {\left( {{2 \over 3} + {1 \over {12}}} \right)} } \right)^{\, - 1} } \right| = {1 \over 4}{4 \over {\sqrt 3 \sqrt 3 }} = {1 \over 3} = \cos \theta 
$$
A: You can also use the Spherical Law of Cosines.
We imagine a sphere centered at $O$, and we take $A$, $B$, $C$ the points where the rays from $O$ meet that sphere as vertices of a curvilinear triangle on the surface of that sphere. The "sides" of that triangle are arcs of great circles, and the lengths of those arcs are directly proportional to the measures of the angles between the rays. (With an appropriate choice of radius, we can say that the lengths are equal to the measures.) The "angles" of the triangle match the dihedral angles between the planes formed by those rays.
Now, since angle $\theta$ is opposite side $\overline{BC}$, we calculate as follows ...

$$\cos\theta = \frac{\cos \angle BOC-\cos \angle AOB \cos \angle AOC}{\sin \angle AOB \sin \angle AOC} = \frac{\cos 90^\circ - \cos 60^\circ \cos 120^\circ}{\sin 60^\circ \sin 120^\circ} \\[20pt]
=\frac{0 - \frac{1}{2}\frac{-1}{2}}{\frac{\sqrt{3}}{2}\frac{\sqrt{3}}{2}} = \frac{1/4}{3/4} = \frac{1}{3}$$  

