1
$\begingroup$

I would like to project from one $2D$ plane onto another. Imagine that I have a picture taken with a camera that was looking onto a plane. Given camera's extrinsic and intrinsic parameters I want to know how the points in the picture map to the points on the pictured plane.

What I know so far is that this is normally achieved using a homography matrix. However, I want to confirm the particular formula for the described projection.

Homography

Let's assume our intrinsic camera matrix is the following: $$ I = \begin{pmatrix}f & 0 & 0 & 0\\ 0 & f & 0 & 0\\ 0 & 0 & 1 & 0\end{pmatrix} $$

The entrinsic matrix describes the position $(x_t, y_t, z_t)$ and rotation of the camera w.r.t. the world coordinates, for demonstration purposes let's assume it's only rotated around $x$ axis: $$ E = \begin{pmatrix}1 & 0 & 0 & x_t\\ 0 & cos\theta & -sin\theta & y_t \\ 0 & sin\theta & cos\theta & z_t \\ 0 & 0 & 0 & 1\end{pmatrix} $$

So if we want to find the projection of a $3D$ point in the world $(x_w, y_w, z_w)$ onto our camera plane, we can now use the final camera matrix to perform the projection transformation: $$ \begin{pmatrix}x_c \\ y_c \\ w \end{pmatrix} = I E \begin{pmatrix}x_w \\ y_w \\ z_w \\ 1\end{pmatrix} $$

And the position on the camera plane (image), currently assuming that image center is at $(0, 0)$, is given by: $x = x_c/w$ and $y = y_c/w$.

Now to get to the original problem and my question ($2D$ to $2D$ plane projection, rather than $3D$ to $2D$ projection) I would do something like the following. First, I only have the location on the image $(x_c, y_c)$ and I want to derive coordinates $(x_w, y_w)$ on a plane in the world. I can rewrite my equation like this: $$ H = IE $$ $$ H = \begin{pmatrix}h_{11} & h_{12} & h_{13} & h_{14}\\h_{21} & h_{22} & h_{23} & h_{24}\\h_{31} & h_{32} & h_{33} & h_{34}\end{pmatrix} $$ $$ \begin{pmatrix}x_c \\ y_c \\ w \end{pmatrix} = \begin{pmatrix}h_{11} & h_{12} & h_{13} & h_{14}\\h_{21} & h_{22} & h_{23} & h_{24}\\h_{31} & h_{32} & h_{33} & h_{34}\end{pmatrix} \begin{pmatrix}x_w \\ y_w \\ 0 \\ 1\end{pmatrix} $$ $$ \begin{pmatrix}x_c \\ y_c \\ w \end{pmatrix} = \begin{pmatrix}h_{11} & h_{12} & h_{14}\\h_{21} & h_{22} & h_{24}\\h_{31} & h_{32} & h_{34}\end{pmatrix} \begin{pmatrix}x_w \\ y_w \\ 1\end{pmatrix} $$ $$ H' = \begin{pmatrix}h_{11} & h_{12} & h_{14}\\h_{21} & h_{22} & h_{24}\\h_{31} & h_{32} & h_{34}\end{pmatrix} $$ $$ H'^{-1} = H'^T $$ $$ \begin{pmatrix}x_w \\ y_w \\ w\end{pmatrix} = H'^T \begin{pmatrix}x_c \\ y_c \\ 1 \end{pmatrix} $$

I would then use $x=x_w/w$ and $y=y_w/w$ as coordinates relative to the $2D$ plane in the world. Is that correct or at least going in the right direction?

Side note: this has been briefly touched upon, but without any good mathematical grounding in someone else's practical question https://stackoverflow.com/questions/20445147/transform-image-using-roll-pitch-yaw-angles-image-rectification and I'm interested in the mathematical foundation of a similar problem.

$\endgroup$
  • $\begingroup$ It seems like you’re on the right track. Note that there are really only $8$ degrees of freedom in a planar perspective transformation matrix. Since a non-zero scalar multiple of a homogeneous coordinate vector represents the same point, a non-zero scalar multiple of a transformation matrix represents the same transformation. This is useful when trying to compute the planar projective matrix that maps a convex quadrilateral onto another. $\endgroup$ – amd Mar 4 '16 at 20:50
  • $\begingroup$ photo.stackexchange.com/questions/6111/… may or may not be helpful. $\endgroup$ – barrycarter Mar 5 '16 at 0:55
  • 1
    $\begingroup$ A couple of corrections to your terminology: the world-to-image mapping is not generally called a homography, which is more usually reserved for a mapping between spaces of the same dimension. Instead, you have a projection matrix that maps from 3-D space to the image, and you’re looking for a homography between the image and an arbitrary plane in the world. Also, $\mathbf t=(x_t, y_t, z_t)$ is not the position of the camera (a.k.a. camera center) but the result of applying the rotation to the camera’s position. The camera center is at $-R\mathbf t$ in world coordinates. $\endgroup$ – amd May 31 '18 at 7:12
2
$\begingroup$

You were doing well up until the line $H'^{-1} = H'^T$. There’s no particular reason to believe that $H$ is orthogonal; quite the opposite, in fact. (I’m going to switch here to the more conventional name $\mathtt P$ for the projection matrix and also make use of the block decomposition $\mathtt P = \left[\mathtt M \mid \mathbf p_4\right]$.) The first two columns of $\mathtt P$, $\mathbf p_1$ and $\mathbf p_2$, are the vanishing points of the world $x$- and $y$ axes. Unless the camera happens to be positioned just right relative to the $x$-$y$ plane, these vectors will not be orthogonal in the image. The upshot is that you have to compute the inverse of $\begin{bmatrix}\mathbf p_1 & \mathbf p_2 & \mathbf p_4\end{bmatrix}$, not its transpose.

This leads to the next potential problem: this matrix might not be invertible. The submatrix $\mathtt M$ is effectively the composition of a rotation and nonsingular affine transformation, so we know the first three columns are linearly independent, but it’s entirely possible that $\mathbf p_4 = \lambda \mathbf p_1 + \mu \mathbf p_2$, that is, that the image of the world origin lies on the vanishing line of the $x$-$y$ plane. This occurs when the camera center lies on this plane. In the specific construction in your question, this isn’t really an issue since you’re reprojecting onto the $x$-$y$ plane, but keep this in mind when generalizing to other planes.

Thus, the reprojection onto the $x$-$y$ plane is given by the homography matrix $$\mathtt H = \begin{bmatrix}\mathbf p_1 & \mathbf p_2 & \mathbf p_4\end{bmatrix}^{-1}.$$ This can be generalized to any plane that doesn’t contain the camera center by inserting an appropriate coordinate transformation $\mathtt B$ into the cascade, i.e., start with $$w\begin{bmatrix}x_c \\ y_c \\ 1 \end{bmatrix} = \mathtt{PB}\begin{bmatrix}x_w\\y_w\\0\\1\end{bmatrix}.$$

For comparison, here’s a construction that makes direct use of back-mapping the image points. Given a plane with homogeneous coordinate vector $\mathbf\Pi$, its intersection with the line through a fixed point $\mathbf C$ not on the plane and an arbitrary point $\mathbf X$ is $\left(\mathbf\Pi^T\mathbf X\right) \mathbf C - \left(\mathbf C^T\mathbf\Pi\right) \mathbf X$. The world coordinates of the camera center can be recovered from $\mathbf P$: its inhomogeneous Cartesian coordinates are $\tilde{\mathbf C}=\mathtt M^{-1}\mathbf p_4$. Finally, an image point $\mathbf x$ back-projects to a ray that intersects the plane at infinity at $\left((\mathtt M^{-1}\mathbf x)^T,0\right)^T$. Putting this all together, and adding a final matrix $\mathtt B$ that imposes a coordinate system on the plane, we get $$\mathtt H = \mathtt B \left(\mathbf C \mathbf \Pi^T-\mathbf C^T \mathbf \Pi \mathtt I_4 \right) \begin{bmatrix}\mathtt M^{-1}\\\mathbf 0^T\end{bmatrix}.$$ With $\mathbf\Pi=(0,0,1,0)^T$—the $x$-$y$ plane—and $$\mathtt B = \begin{bmatrix}1&0&0&0\\0&1&0&0\\0&0&0&1\end{bmatrix}$$ this produces the same homography matrix as above.

The details of the anatomy of the projection matrix that I used above can be found in Hartley and Zisserman’s Multiple View Geometry In Computer Vision and other standard references on the subject.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.