I have worked this out on paper and I'm hoping someone can point out my mistake.
A "look at" matrix is supposed to rotate you so your old $(x,y,z)$ coordinate axes become aligned with a new set of coordinate axes $(newx, newy, newz)$. Basically a change of basis.
Working in an LH coordinate system, using row major matrices, according to this document, this rotation matrix should be constructed as:
$$ \left[ \matrix { newx.x&newy.x&newz.x \\ newx.y&newy.y&newz.y \\ newx.z&newy.z&newz.z } \right] $$
(Note I have left off the final translation row from the article - assuming we are working with 3x3 matrices here.)
Assume I pass in
- $eye=(0,0,0)$ (so there is no translation, only rotation of the coordinate axes),
- $at=(0,0,1)$ (so we are "looking down the +z axis"),
- $up=(\frac{-1}{2}, \frac{\sqrt{3}}{2}, 0)$ (so we basically are doing a 30 degree rotation about the z-axis).
Compute
- $newz=(0,0,1)$,
- $newx=( \frac{\sqrt{3}}{2}, \frac{1}{2}, 0 )$,
- $newy=(\frac{-1}{2}, \frac{\sqrt{3}}{2}, 0)$
So far we are on track: the $newz$ vector points in the same direction it did before, $newx$ is 30 degrees rotated about the z-axis, and $newy$ is 30 degrees rotated about the z-axis.
Form the rotation matrix:
$$ \left[ \matrix { \frac{\sqrt{3}}{2} & \frac{-1}{2} & 0 \\ \frac{1}{2} & \frac{\sqrt{3}}{2} & 0 \\ 0 & 0 & 1 } \right] $$
(I tested this on a computer and this is exactly the matrix that the library will form, with these inputs),
Yet premultiply with a sample vector like $ \left[ \matrix{1&0&0} \right] $, and get $ \left[ \matrix{ \frac{\sqrt{3}}{2} & \frac{-1}{2}& 0 } \right] $ as the result, meaning $ \left[ \matrix{1&0&0} \right] $ lands rotated negative 30 degrees, not as expected.
If you transpose this rotation matrix though, you will get the expected answer of $ \left[ \matrix{ \frac{\sqrt{3}}{2} & \frac{1}{2}& 0 } \right] $.
Where have I gone wrong?