Calculate 2d rotation of a wall in a picture given 4 edge points I want to compute the rotation (angle alpha) between the wall and the camera. (see Figure 1) The camera takes a photo as pictured in "camera view", and I get the 4 edge points of the wall (in 2d (x,y) coodinates).
How can I compute alpha if I know that the wall has to be a 16:9 rectangle if not distorted?
 A: I will reformulate the problem, so that I set it on the right footing for myself and you can decide whether it works for you or not. 
We have an observer, placed at point $O$ facing a transparent screen $S$ (like a window), and there is a wall $W$, in the form of a rectangle, behind the screen, i.e. the plane $S$ of the screen lies between the observer $O$ and the rectangular wall $W$. The observer sets up an orthonormal coordinate system $Oxyz$ so that the coordinate plane $x, y$ is parallel to the screen $S$ and the $z$ coordinate line goes from the observer $O$ to the screen $S$ and is perpendicular to $S$. Then, the observer sets up a coordinate system $O_sxy$ on the plane $S$ by simply projecting orthogonally the $Oxy$ coordinate plane onto $S$. Here $O_s$ is the orthogonal projection of $O$ onto $S$, i.e. $O_s$ is the intersection point of $S$ with the $z$ axis and the axes $O_sx$ and $O_sy$ are parallel to the axes $Ox$ and $Oy$ respectively. All of this means that if $h = |OO_s|$ is the distance between $O$ and the plane $S$, then a point with  coordinates $(x,y)$ with respect $O_sxy$ in $S$ has coordinates $(x,y,h)$ in the system $Oxyz$. 
Let $A_w, B_w, C_w, D_w$ be the vertices of the rectangle $W$ and let $A_s$ be the intersection of line $OA_w$ with the screen $S$, let $B_s$ be the intersection of line $OB_w$ with the screen $S$, let $C_s$ be the intersection of line $OC_w$ with the screen $S$ and let $D_s$ be the intersection of line $OD_w$ with the screen $S$. The observer know the exact coordinates of the points $A_s, B_s, C_s, D_s$ in the coordinate system $Oxy$. 
Let us figure out some geometry. Let the pair of planes $A_wB_wO$ and $C_wD_wO$ intersect at their common line we call $p$ and let the pair of planes $B_wC_wO$ and $D_wA_wO$ intersect at their common line we call $q$. Both lines $p$ and $q$ pass trough $O$. Since $A_wB_w$ and $C_wD_w$ are parallel (by assumption $A_wB_wC_wD_w$ is rectangle), the common line of intersection $p$ is parallel to both $A_wB_w$ and $C_wD_w$ and thus to the whole plane $W$. Analogously, the common line of intersection $q$ is parallel to both $B_wC_w$ and $D_wA_w$ and thus to the whole plane $W$. By the way, as $A_wB_w$ and $B_wC_w$ are perpendicular, so are $p$ and $q$, which means they are transverse to each other (they are not the same line).
Let us look at the line $p$ and the plane $S$. Either $p$ intersects $S$ at a unique point called $P_s$, or $p$ is parallel to $S$. 
Case 1. Assume $p$ intersects $S$ at the point $P_s$. Observe that by construction line $p$ also lies on the plane $OA_wB_w$ (line $p$ is the intersection line of planes $OA_wB_w$ and $OC_wD_w$). However, also by construction, line $A_sB_s$ lies on $OA_wB_w$ too (recall $A_sB_s$ is the intersection line between $S$ and $OA_wB_w$) so $A_sB_s$ either intersects the line $p$ or is parallel to it. If $A_sB_s$ intersects $p$, their point of intersection lies on $S$ and on $p$, because $A_sB_s$ lies on $S$. However, there is only one point that lies on both $S$ and $p$ and that is point $P_s$. Therefore, in this case, line $A_sB_s$ passes thorough point $P_s$. If $A_sB_s$ is parallel to $p$ then $p$ must be parallel to $S$ which contradicts the assumption $P_s = p \cap S$. 
Absolutely analogous arguments hold for line $C_sD_s$. Again by construction line $p$ also lies on the plane $OC_wD_w$ (line $p$ is the intersection line of planes $OC_wD_w$ and $OA_wB_w$). However, also by construction, line $C_sD_s$ lies on $OC_wD_w$ too (recall $C_sD_s$ is the intersection line between $S$ and $OC_wD_w$) so line $C_sD_s$ either intersects the line $p$ or is parallel to it. If $C_sD_s$ intersects $p$, their point of intersection lies on $S$ and on $p$, because $C_sD_s$ lies on $S$. However, there is only one point that lies on both $S$ and $p$ and that is point $P_s$. Therefore, in this case line $C_sD_s$ passes thorough point $P_s$. If $C_sD_s$ is parallel to $p$ then $p$ must be parallel to $S$ which contradicts the assumption $P_s = p \cap S$. 
In conclusion, in this case, line $A_sB_s$ line $C_sD_s$ and line $p$ intersect at a common point $P_s$.
Case 2. Assume $p$ is parallel to $S$. Then arguments very similar to the ones  above lead to the conclusion that line $A_sB_s$, line $C_sD_s$ and line $p$ are parallel to each other.  
If we argue the same way about line $q$ and $S$, we will conclude that either line $B_sC_s$, line $D_sA_s$ and line $q$ intersect at a common point $Q_s$ or they are parallel to each other.
Now denote by $W_o$ the plane spanned by lines $p$ and $q$. This plane passes through point $O$ and since $p$ and $q$ are parallel to $W$, so is the plane $W_o$ spanned by them. Thus, the angle between $S$ and $W$ you are looking for is in fact equal to the angle between $S$ and $W_o$. But as already proved, in the general case where the lines $p$ and $q$ are not parallel to the screen $S$, the plane $W_o$ passes through the points $P_s \in p$ and $Q_s \in q$, where $P_s$ is the intersection point of lines $A_sB_s$ and $C_sD_s$, and $Q_s$ is the intersection point of lines $B_sC_s$ and $D_sA_s$. So the plane $W_o$ is also spanned by the three points $P_s, Q_s$ and $O$ and these are points you can find!
Here is now the algorithm one can use. You are given the coordinates of the four points $A_s, B_s, C_s, D_s$ with respect to the coordinate system $O_sxy$ forming a convex quadrilateral $A_sB_sC_sD_s$ in the plane $S$ (labels are in consecutive cyclic order). 
Step 1. Given the points $A_s$ and $B_s$, find the line $A_sB_s$. 
Step 2. Given the points $C_s$ and $D_s$, find the line $C_sD_s$. 
Step 3. Find the point of intersection $P_s = (x_P,y_P)$ of lines $A_sB_s$ and $C_sD_s$ (assuming the lines are not parallel). 
Step 4. Find the point of intersection $Q_s = (x_Q,y_Q)$ of lines $B_sC_s$ and $D_sA_s$ (assuming the lines are not parallel). 
Step 5. Form the three dimensional vectors $\overrightarrow{OP_s} = (x_P, y_P, h)$ and $\overrightarrow{OQ_s} = (x_Q, y_Q, h)$  
Step 6. Compute the cross product $\overrightarrow{OP_s} \times \overrightarrow{OQ_s}$ 
Step 7. Compute the length $|\overrightarrow{OP_s} \times \overrightarrow{OQ_s}|$. Normalize the vector $\frac{\overrightarrow{OP_s} \times \overrightarrow{OQ_s}}{|\overrightarrow{OP_s} \times \overrightarrow{OQ_s}|}$
Step 8. Let $\overrightarrow{e_3} = (0,0,1)$. Then the dot product $$\cos{\alpha} = \left(\frac{\overrightarrow{OP_s} \times \overrightarrow{OQ_s}}{|\overrightarrow{OP_s} \times \overrightarrow{OQ_s}|} \,\cdot \, \overrightarrow{e_3} \right)$$ defines the cosine of your angle $\alpha$. 
The case when $A_sB_s$ and $C_sD_s$ are parallel can be handled analogously, by simply replacing $\overrightarrow{OP_s}$ by $\overrightarrow{A_sB_s}$. The case when $B_sC_s$ and $D_sA_s$ are parallel can be handled, by simply replacing $\overrightarrow{OQ_s}$ by $\overrightarrow{B_sC_s}$. If both pairs of lines are parallel, then $\alpha=0$ because the screen $S$ is parallel to the wall $W$. 
A: You can see below a top view of your wall, with $B$ and $C$ the midpoints of the vertical walls, projected onto the plane of view at $B'$ and $C'$. I'm assuming that line $VA$, joining the camera to the center of the wall, is perpendicular to $B'C'$.
Let $VA=d$ and $VA'=s$. You have $BM=AB\sin\alpha$ and $AM=AB\cos\alpha$. By similarity we then obtain:
$A'B':MB=(VA-AM):VA'$, whence:
$$
A'B'={s\sin\alpha\over d-AB\cos\alpha}AB
\quad\hbox{and, analogously:}\quad
A'C'={s\sin\alpha\over d+AC\cos\alpha}AC.
$$
Summing up these equalities we thus get:
$$
\tag{1}
B'C'={sd\sin\alpha\over d^2-a^2\cos^2\alpha}BC,
$$
where I set $a=AB=AC$.

If $DE$ (with midpoint $B$) and $FG$ (with midpoint $C$) are the vertical sides of the wall (see picture below), their projections on the plane of view can be computed in a similar way because we have for instance $DE:D'E'=VB:VB'=VM:VA'$, which gives:
$$
D'E'={s\over d-a\cos\alpha}DE
\quad\hbox{and, in a similar way:}\quad
F'G'={s\over d+a\cos\alpha}FG,
$$
so that
$$
\tag{2}
D'E'+F'G'={sd\over d^2-a^2\cos^2\alpha}(DE+FG).
$$

Dividing $(1)$ by $(2)$ we finally get:
$$
{B'C'\over D'E'+F'G'}=\sin\alpha{BC\over DE+FG}.
$$
As $DE/BC=FG/BC=9/16$, from the above relation we easily obtain
$$
\sin\alpha={9\over 8}{B'C'\over D'E'+F'G'}.
$$
