I have a question on how best to "average" transformation matrices. Say that I have n number of 4x4 transformation matrices, and I wanted to find a matrix that approximated each one of the n 4x4 transformation matrices (an average of sorts). Are there any methods that would work?
Unfortunately, there's no really good way to average such transformations. In particular, in three dimensions there is no possible averaging operation $\mathcal A$ with all of the following natural and desirable properties:
$\mathcal A$ is symmetric -- that is, $\mathcal A(M_1,M_2)=\mathcal A(M_2,M_1)$ for all $M_1$ and $M_2$.
$\mathcal A$ is invariant under rotations of the coordinate system.
Whenever the inputs to $\mathcal A$ are both rotation matrices (or invertible or has determinant 1), the output is a also a rotation matrix (or invertible or has determinant 1).
So at least one of these properties has to be given up. The only one that it really makes any sense to do without is (2), but even so the resulting outcome is going to be discontinuous and rather non-intuitive.
The question is a bit vague, but one way to go about this is to parametrize your transformation matrices using a Euclidean structure. The translation part of the transformation is already Euclidean. The problem is with the rotation.
One good Euclidean represetntation of SE(3) is the Lie algebra. You could convert all matrices into 6x1 vectors using the Lie algebra (and extracting the twist), then compute the average in this 6-space. Finally, transform the average back to SE(3) via the exponential map.
See here for general information on the exponential map http://en.wikipedia.org/wiki/Exponential_map_%28Lie_theory%29
and here for more specific information about rigid-body transformations and code http://www.mathworks.com/matlabcentral/fx_files/24589/1/content/kinematics/doc/index.htm
I would first pick a distance function between transformations. You'll have to combine a translational part and a (harder to pick) rotational part. Here is an article discussing distances on SO(3): http://ai2-s2-pdfs.s3.amazonaws.com/5617/8de1001efe54792ad93f6980de5d5e91906b.pdf
If you think one of the transformations in your set is a reasonable center for your purposes, then you can pick the point that minimizes the sum absolute (or squared) distance to the other points. You can also do a form of sample and consensus, where your objective is essentially the number of other points within a fixed radius ball.
If not, then you can set up a constrained nonlinear optimization of the function you defined, and hope for the best. I think the ceres solver can constrain quaternions, for example...
There's a trick I've used which is mathematically a little dodgy but gives the desired result for a certain set of use cases (particularly where you want to avoid 'pops' when your rotations are continuously changing).
First split the matrices so you have lists of positions, forward vectors and up vectors.
Average each of the lists of vectors.
Normalize the forward and up vectors and use them to construct a rotation (favouring the forward vector and just using the up to calculate roll around the forward vector).
Then just reconstruct your matrix.
Again, the average isn't the true average but it's a good approximation and gives a nice continuous result (unless your average forward or up results in a vector of exactly zero length).
It also lets you weight the values as you can just multiply each position/forward/up by the weight, add them up and divide by the total.