Calculate object position in a perspective view I am trying to detect a court field from a video stream via machine learning.
In my scenario I have a court field with known dimensions:

*

*width: 10m

*height: 20m (10m per field)

*height of the net: 0,88m


I am already able to detect the upper and lower bounds of the court, as well as the top edge of the net. As the bottom edge of the net contains no usable visual cues, I am trying to calculate the bottom edge based on the known dimensions.
In the following picture you can see the detected lines in black. I want to calculate the red line which maps to the middle line of the court.

The perspective is not fixed, as the perspective might slightly change in the video stream or between streams.
Thanks in advance for your input!
 A: If you know (or assume) that the court is a regular tennis court
(rectangular with a net exactly halfway between the ends),
the point at which the diagonals of the court cross each other is directly under the net.
The perspective images of the diagonals therefore also cross each other at the red line in your diagram.
The images of straight lines are straight, so you just need to work out where the diagonals of the trapezoid meet and put your red line through that point parallel to the bases of the trapezoid.
A similar technique works if the court is viewed from a different angle so that the ends no longer appear parallel. Find the intersection of the two diagonals of the quadrilateral, find the vanishing point of the two ends of the court (that is, the point of intersection if you continue those lines indefinitely), and make the red line lie along the infinite line through those two intersections.
This is a classic artist's technique for drawing a tiled floor.
A: Note that the red line is in the same plane as the rest of the court. Use the top line and the bottom line. They form a trapezoid. The red line is the line parallel to the bases at half height, so the coordinates of the points will be the midpoints of the slanted sides.
A: Assign a coordinate system attached to the playing field with its origin at the center point of the field, and with its $x$-axis extending to the right, while its $y$-axis pointing from near to far, and the $z$ axis pointing upward from the ground.
We'll take the following three points on the field, whose images are known: $P_1 = (5, 10, 0), P_2 = (5, -10, 0), P_3 = (5, 0, 0.88)$
Let their images be $Q_1 =(x_1, y_1), Q_2 = (x_2,y_2), Q_3 = (x_3, y_3)$ where $x_1, y_1, x_2, y_2, x_3, y_3 $ are all known.
Now, let the camera be centered at  $C = (0, -a, h)$, with $a, h$ unknown yet.
And suppose the normal vector of the projection plane is pointing in the direction:
$n = (0, \cos \theta, -\sin \theta) $
and let the projection plane be a distance $z_0$ from the camera center.
We're going to generate a local coordinate system and attach it to the camera center
$x' = (1, 0, 0)$
$y' = -  n \times x' = (0 , \sin \theta, \cos \theta)$
note that this coordinate system is left-handed.
Next, we'll find the coordinates of the vectors $P1, P2, P3$ with respect to the camera
The relation between the world coordinates and the local frame coordinates is
$ P = R p + C $
where
$R = \begin{bmatrix} 1 && 0 && 0 \\ 0 && s && c \\ 0 && c &&  -s \end{bmatrix}$
and $s = \sin \theta, c = \cos \theta, and $ C$ is specified above.
Now, $R^{-1} = R^T = R $
Therefore, $p = R^{-1} (P - C) = R (P - C) $
Therefore, the local coordinates of $P_1$ are
$p_1 = R  (5 , 10 + a, - h )  = (5, s (10 + a) - c h , c (10 + a) + s h )$
Now we have to scale this vector down by a scale factor $\alpha$, such that   $\alpha ( c (10+a) + s h) = z_0$
From this we deduce that the coordinates of the image of $P_1$ are given by
$ Q_1=(x_1, y_1) = \dfrac{ z_0 }{ c(10 + a) + s h } (5, s (10 + a) - c h )$
Similarly we can derive the expressions for $Q_2$:
$Q_2 = (x_2, y_2) = \dfrac{z_0} {c (-10 + a) + s h} (5, s(-10+a) - c h ) $
There are four unknowns in the above equations, which are $a, h, z_0, \theta$ and there are four equations obtained from the $2$ components of each of the two points.  So we can solve for the unknowns, numerically.  Once this is done, we can obtain the image of $P_4 = (5, 0, 0) $ which corresponds to the bottom of the net.
Note that we don't need the point $P_3$ in our calculations.  Also note that, all the above assumes that the camera lies in the central plane of the court dividing it to left and right.
If, however, the image coordinates have been scaled uniformly (i.e. in both the $x'$ and $y'$ directions) by an unknown scale factor $S$ and shifted in the $y'$ direction by an unknown quantity $F$, then can still solve for $a, h, (S z_0), \theta, F$ , if we consider the image coordinates $Q_3$ of the tip of the net.
