# texture mapping from a camera image to a 3D surface acquired by a kinect

I have the following problem:

A kinect camera capture a 3D surface and save it as a .obj files containing all the positions of the vertices (in the kinect coordinate system).

If I take a picture from the kinect, I can successfully apply that picture on the 3D surface. (see my previous question here)

Now, I want to do the same thing, except the picture will be taken from a different camera (a webcam) placed on the kinect.

As you can see from my previous question, I only needed the camera pose, the X Y Z point and the intrinsic matrix.

I have the intrinsic matrix from the webcam, I still have access to the X Y Z point (thanks to the .obj file), and I have the pose of the kinect.

Since I have the kinect pose, I needed to find a transformation matrix to go from the kinect pose to the webcam pose. (the pose is described by a rotation matrix and a translation vector)

In order to do so, I determined the pose of the webcam and the kinect individually in a common coordinate system (see the answer here for more details), I used millimeters as units because I believe the kinectfusin toolkit uses them as well.

So now, I have two Rotation|Translation matrices, one describing the pose of the kinect and the other describing the pose of the webcam. Btw, I guess I did it right because the translation vectors seem correct when I measure the distance between the cameras (with some errors of some millimeters tho)

After some testing I found that openCV and kinectfusion use different coordinate systems

So in order for the openCV coordinate system to match the kinectfusion one, I just multiply the Z coordinate of the translation vectors by -1 (Should I do something about the rotation matrices ??)

Then, to get the transformation I applied the following formulas:

relative_translation_vector = webcam_translation_vector - kinect_translation_vector relative_rotation_vector = webcam_rotation_matrix * inverse(kinect_rotation_matrix)

I then multiplied the R|T matrix from the kinect with the R|T matrix representing the relative transformation and applied the following formula (see my previous question here again):

Well...I thought that would be good enough, but unfortunately my texture is not projected on the correct location..so I guess somewhere in the maths I got something wrong...

It's been many days I'm trying to figure out the problem but I can't find what my mistake is. Any help or suggestions will be greatly appreciated ! Thank you