I am working with a Nao robot. One of the things I want to do is the following:

  • Take an image of geometrical objects in front of the Nao (done)
  • Extract features from the objects in the image, such as x, y, color, etc.. (done)
  • Make the Nao point to one of the objects (TO DO)

So, what I need is to transform a 2D coordinate in the image plane to a 3D coordinate in the robot's coordinate system. I have some idea on how to do this, but I am not sure if this is correct.

So, I start by transforming the 2D image coordinate to the 3D coordinate system that originates in the camera of the Nao. For this, I use height and width (in pixels) of the image ($r_w$ and $r_h$) and the horizontal and vertical camera opening ($\theta_v$ and $\theta_h$). This transformation is: $$ v_c = \begin{bmatrix} 1 \\ -\frac{x_i}{r_h} \tan{\frac{\theta_h}{2}} \\ \frac{y_i}{r_w} \tan{\frac{\theta_v}{2}} \end{bmatrix} $$

Then, I need to translate this vector using the orientation of the head of the Nao. The resulting coordinate system is parallel to the entire robot's coordinate system. The rotation of the camera is given by $R_c$. The translation is: $$ v_t = R_c v_c $$

Finally, I project this coordinate system to the robot's coordinate system. The Nao can use 2 coordinate system, one that originates on the ground between its legs (FRAME_ROBOT) or one that originates in its chest (FRAME_TORSO). Which one is used has no real importance for me. Let's say the offset between the coordinate systems is given by $t_c$. The transformation is: $$ v_r = v_t + t_c $$

So, given a position in the image $(x_i, y_i)$, I get this position in the robot's coordinate system. Is this a correct approach?

Original question on Robotics Stack Exchange


1 Answer 1


Unfortunately, it's impossible to reconstruct 3D position of the object. You can check the fact by trying to touch a specific location of an object only by using one eye. This is because observation causes degeneration of dimension.

I suggest using two cameras equipped with Nao to acquire depth information. As a preprocess, you have to get intrinsic & extrinsic camera parameters including focal length, optical center and relative posture of cameras. This process is called calibration. OpenCV provides useful toolsets to get accurate camera parameters. Once the cameras are calibrated, you can use triangulation to determine their 3D position in camera's coordinate. Note that triangulation can also be done within OpenCV. All you have to do there is coordinate transformation from camera's coordinate to robot's coordinate or something.

  • $\begingroup$ Although this answer is correct (that it is impossible to reconstruct), actually you don't need complete information about 3D position to move the robot to that position. Just which direction is the point in with relative to the robot. $\endgroup$
    – user202729
    Oct 28, 2017 at 12:48
  • $\begingroup$ The Nao does have a second camera. There is one on the forehead and one in the mouth. Maybe I can look into this for triangulation. Also, the objects I show the Nao are always approximately the same size. So I could also hard-code information about these objects. However, I don’t want to know how far the object is, but the coordinates of the object’s center in the robot’s coordinate system. Wouldn’t my method give a reasonable approximation? Or is it just wrong? $\endgroup$
    – JNevens
    Oct 28, 2017 at 12:50
  • $\begingroup$ Ah... I might have misunderstood the question. I supposed JNevens wanted to move e.g. Nao's finger to point the location. You mean you want Nao to look at the point? $\endgroup$
    – mhirano
    Oct 28, 2017 at 12:53
  • $\begingroup$ Yes, I want Nao to point to the object. $\endgroup$
    – JNevens
    Oct 28, 2017 at 12:54
  • $\begingroup$ If you just want Nao to look at the object, you might not need to locate the coordinate of the object. Simply you can compute tilting angles derived from camera's intrinsic parameters. $\endgroup$
    – mhirano
    Oct 28, 2017 at 13:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .