Given the problem of trying to find a transformation matrix from one camera to another on a vehicle in a calibration room with the usual checkerboard floor and walls, we can capture images from both cameras and label each vertex with the physical and screen coordinates.
Now, the code I have inherited uses the OpenCV library API
cv::solve(... cv::DECOMP_SVD) to try to find a transformation matrix that transforms points from camera 1's coordinate space to camera 2's. Currently the code works reasonably well and usually produces a good result, but I have a couple of questions.
Should we solve with SVD for each plane (floor and each wall) separately, or should we solve all at once? (Currently the code does the second)
Given that there might be few if any points actually visible in both cameras simultaneously, but we have the camera's $K$ and $D$, and have calculated the $R$ and $T$ for each image, how accurate is it to extrapolate well outside the FOV of a camera?