I have the following problem:
A kinect camera capture a 3D surface and save it as a .obj files containing all the positions of the vertices (in the kinect coordinate system).
If I take a picture from the kinect, I can successfully apply that picture on the 3D surface. (see my previous question here)
Now, I want to do the same thing, except the picture will be taken from a different camera (a webcam) placed on the kinect.
As you can see from my previous question, I only needed the camera pose, the X Y Z point and the intrinsic matrix.
I have the intrinsic matrix from the webcam, I still have access to the X Y Z point (thanks to the .obj file), and I have the pose of the kinect.
Since I have the kinect pose, I needed to find a transformation matrix to go from the kinect pose to the webcam pose. (the pose is described by a rotation matrix and a translation vector)
In order to do so, I determined the pose of the webcam and the kinect individually in a common coordinate system (see the answer here for more details), I used millimeters as units because I believe the kinectfusin toolkit uses them as well.
So now, I have two Rotation|Translation matrices, one describing the pose of the kinect and the other describing the pose of the webcam. Btw, I guess I did it right because the translation vectors seem correct when I measure the distance between the cameras (with some errors of some millimeters tho)
After some testing I found that openCV and kinectfusion use different coordinate systems
So in order for the openCV coordinate system to match the kinectfusion one, I just multiply the Z coordinate of the translation vectors by -1 (Should I do something about the rotation matrices ??)
Then, to get the transformation I applied the following formulas:
relative_translation_vector = webcam_translation_vector - kinect_translation_vector relative_rotation_vector = webcam_rotation_matrix * inverse(kinect_rotation_matrix)
I then multiplied the R|T matrix from the kinect with the R|T matrix representing the relative transformation and applied the following formula (see my previous question here again):
Well...I thought that would be good enough, but unfortunately my texture is not projected on the correct location..so I guess somewhere in the maths I got something wrong...
It's been many days I'm trying to figure out the problem but I can't find what my mistake is. Any help or suggestions will be greatly appreciated ! Thank you