# Resolving the up-vector of an image through multiple transforms

First up, a disclaimer. I'm an engineer not a mathematician, so my approach thus-far may be a little ham-fisted. I have however reached a point in this problem where my usual engineering references aren't helping, and since I'm phrasing this as a vector and co-ordinate frame problem I figured I'd try asking here.

The problem is as follows. I have a pan-tilt system which points a camera in a specified direction (azimuth and elevation) relative to it's body frame. We define the pointing vector of the camera to be a unit vector in the body frame pointing along the desired azimuth and elevation,. The up vector for the image is therefore a vector perpendicular to the pointing vector which is co-planar with the Z (up) axis of the body frame.

We have a intertial unit on the pan-tilt system which gives us the roll-pitch-yaw rotations between the body frame and a global reference frame. The main task is to, when given an azimuth-elevation pointing angle command in the global frame, calculate the required Az and El for the pan-tilt unit in the body frame.

This enables us to keep the camera pointing in the same global direction as the vehicle carrying the sensor pitches, rolls and yaws underneath us, and is working just fine. The problem is that we are not yet accounting for any rotation of the unit about the pointing vector, and therefore the image from the camera appears to roll as the platform moves.

My approach to solve this problem is to attempt the following:

1. Translate the global up-vector into the body frame of the pan-tilt unit (trivial).
2. Find the projection of that translated global up-vector in the image plane (the perpendicular plane to the pointing vector)
3. Find the angle between the image up-vector and the projected global up-vector.
4. De-rotate the image in software by this angle. (trivial)

My problems are two-fold. Firstly, I can't figure out how to find the projection of the translated global up vector in the image plane, or even whether such a thing is meaningful without some further definition of a co-ordinate frame on that image plane. Secondly, this algorithm needs to be written to be extremely computationally efficient. I need to be able to calculate this parameter on a lightweight microcontroller at a rate upwards of 200Hz.

So, if there is some standard solution which I'm missing, please point me in the direction of a good reference so I can try and get my head around what is actually going on. Alternatively, if anyone can spot a simpler approach to this one then I'd absolutely love to hear it.

Thanks for the help from a seriously confused engineer.....

-

Your approach sounds good. To project the up vector onto the image plane, subtract its projection onto the pointing vector from it; according to Wikipedia this is also called the “vector rejection”. Its formula is

$$\mathbf{a} - (\mathbf{a}\cdot\mathbf{\hat b})\mathbf{\hat b}$$

where $\mathbf{a}$ is the up vector and $\mathbf{\hat b}$ is the (normalized) pointing vector.

As to efficiency: have you considered recording the vectors (inertial data and pointing vector) instead of the calculated angle? That would give you more flexibility in postprocessing (not discarding information) as well as avoiding doing trigonometry on your microcontroller, provided you can store/transmit the higher data rate — I assume there's a more powerful computer doing the image rotation for which this calculation would be peanuts.

-
Cheers for that Kevin, that vector rejection looks like exactly what I'm after. I actually came up with a solution over the Easter break, but I'll have to compare mine and yours to see which is more efficient. With regards to performing the rotation on the PC, this transform is part of a product we sell, and I'm trying to minimise the amount of external code required. If I absolutely have to do it on the PC, then I can accept that, but I'd much rather have all the work done on the Micro if possible. –  OcularProgrammer Apr 9 '12 at 22:28