Let's say we have a scenario like this:

There is the pink camera and the blue camera. Both cameras have different fields of view and orientations. The pink camera takes a picture like this one:

enter image description here

On this 2D image that the pink camera captured, I know exactly the coordinates of the center of the green circle. Is it possible to then calculate an approximation of the orientation that the blue camera should be in to be pointing directly at the green circle? How accurate can that be?

To summarize, this is the data we have:

  • The position of each camera
  • Both camera's field of view size and proportion
  • The orientation that the pink camera was in when it captured the image
  • The location of the center of the green circle inside the image

Edit: I should make clear that what I'm asking for is an approximation, specifically on the assumption that we would be considering the surface as just a plane, all the objects have a very insignificant heights. The heights on the illustrations are just for a better visualization, including the green sphere. I apologize for not making this more clear before.

  • $\begingroup$ Try making it 2D and see how that helps. $\endgroup$ – weareallin Apr 7 '19 at 23:28
  • 1
    $\begingroup$ Do you know the location of the gray plane and the distance of the dot above the plane? $\endgroup$ – David K Apr 8 '19 at 11:50
  • $\begingroup$ @DavidK yes, and no, respectively. But consider that the distance of the dot is very minimal, so treating the distance as 0 would give a good enough approximation. $\endgroup$ – Felipe Müller Apr 8 '19 at 16:26

As John Hughes explained in his answer, what you want to do is not possible given only the information that you first provided in your question. However, with the additional information about the plane that the sphere is sitting on and the assumption that it is small relative to its distance from the cameras, the problem becomes soluble. Basically, you just need to compute the intersection of the back-projection of an image point with that plane.

Here’s one way to do this: From the information that you have about the pink camera, you’ll need to work out its $3\times4$ projection matrix $\mathtt P = [\mathtt M\mid\mathbf p_4]$. Working in homogeneous coordinates, you know two points on the back-projection of an image point $\mathbf x$: the camera’s center $\mathbf C$ and $\mathtt P^+\mathbf x$, where $\mathtt P^+$ is the right pseudo-inverse of $\mathtt P$: $$\mathtt P^+=\mathtt P^T(\mathtt P\mathtt P^T)^{-1}.$$ If the “ground” plane has equation $ax+by+cz+d=0$, then setting $\Pi=(a,b,c,d)^T$, the intersection of the back-projected ray and this plane is $$((\mathtt P^+\mathbf x)^T\Pi)\mathbf C-(\mathbf C^T\Pi)(\mathtt P^+\mathbf x). \tag1$$ If you’re not comfortable working with homogeneous coordinates, you can instead compute the intersection of the ground plane with the line through the pink camera center that has direction vector $\mathtt M^{-1}\mathbf x$.

Once you have the coordinates of this intersection point, working out the direction in which to point the blue camera is a simple exercise. It won’t point directly at the center of the green sphere, but if your assumption that it is small relative to its distance from the cameras, the error will also be small.

  • $\begingroup$ I found this result interesting: "the line through the pink camera center that has direction vector $M^{−1}x$". I know that we can get the direction of the ray back-projected from a point $x$ using the calibration matrix $K$ to get $d = K^{-1}x$. I did not know that $M^{-1}x$ also gives the same direction. Could you please point me to a proof of this statement? $\endgroup$ – ImmenselyHappy May 30 '20 at 18:02
  • 1
    $\begingroup$ @ImmenselyHappy If you have in mind the usual $\mathtt P=\mathtt K[\mathtt R\mid\mathbf t]$ decomposition, then $\mathtt K^{-1}\mathbf x$ gets you the back-projection direction in the camera coordinate system. It still needs to be rotated to get world coordinates, and since $\mathtt M=\mathtt{KR}$... Note that this only works for finite cameras: $\mathtt M$ is singular for an affine camera (camera at infinity). OTOH, back-projecting via $\mathtt P^+$ always works. See Hartley & Zisserman or most other standard references. $\endgroup$ – amd May 30 '20 at 21:22

Consider the following:

Camera 1: looks towards origin from $(20,0,0)$

Camera 2: Looks toward origin from $(0, 20, 0)$; the image of the $x$-axis of 3-space becomes, in image $2$, the $x$-axis of image 2.

They have the same field of view, etc.

We place (in 3D) a green sphere which then appears as a green dot at the center of the camera 1 image. The location of this green sphere in 3D is at $(x, 0, 0)$ for some $x$. That's all you can know from the information you have about camera $1$.

The position of the dot in the camera 2 image is ... somewhere along the $x$-axis of that image. It could be anywhere.

So from the information given, the answer is "no, you cannot determine the orientation of camera 2 that will place the green dot at the image center," because you do not know the "depth" of the green dot (relative to camera 1), which is essential for determining the green sphere's image location in camera 2.

  • $\begingroup$ If you look in the image the depth of the green sphere is very minimal, so if we assumed the depth as being zero, I believe it would result in a good approximation? $\endgroup$ – Felipe Müller Apr 8 '19 at 16:30
  • 1
    $\begingroup$ By "depth", I mean "distance along the viewing direction." If you assume that's zero, then the green dot is exactly at the position of camera 1, which surely is a very bad solution to the general problem you asked about. My example shows that in a pretty simple situation, there's a free parameter -- a whole line of possible solutions. That situation should make you think "Oh, darn, I need more data," not "I should assume some random thing and maybe it'll produce a good-enough answer." $\endgroup$ – John Hughes Apr 8 '19 at 17:03
  • $\begingroup$ Based on the comments, the location of the gray plane is an additional piece of information that is available (though not mentioned in the initial posting of the question) and the green dot is close to that plane. So there's the (approximate) depth of the green dot. $\endgroup$ – David K Apr 8 '19 at 21:24
  • $\begingroup$ Sigh. Well, I eave it to the OP to work out an answer then, perhaps with the same quality with which the question itself was presented. $\endgroup$ – John Hughes Apr 8 '19 at 22:30

You can calculate the center of the blue camera's view and make it equal to the center of green sphere. If it's not achievable in your program you also can evaluate the center of the blue camera's view and change the heights or tilt angle of the blue camera until the center values match.
Guess I understood the problem further. If we create 3d coordinate system the coordinates of one dimension must be the same for centers to match, so that we can create a 2d diagram like so: enter image description here Thus the Z coordinates of sphere and camera are equal. From here you can evaluate the angle A (or whatever you need) to point camera on the right place. Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.