Normalizing 2D coordinates that are arbitrarily transformed. This was originally posted as a programming question so I will try to keep it as abstract as possible.
We are dealing with two images. The first is the original and second is a distorted version of the original. Sample images attached.
The original has one square (marker) drawn at the top-left, one at the top-right and one at the bottom right. The three squares are used for determining orientation. We know the $width$, $height$ of the image and the $(x,y)$ coordinates of all three markers. There is also a point of interest $(p)$ on the image whose $(x,y)$ position we know.
The distorted version has some transformations applied to it (translation, scaling and rotation only). This is very common if a document/image is printed and scanned back into the system. It could even be upside down, skewed etc. but for now let's ignore that and assume that even rotation is no more than 15 degrees.
Now, I am able to detect the $(x,y)$ coordinates of all markers and calculate the angle of rotation, the amount of $(x,y)$ translation and the $(x,y)$ scaling factor.
Now, I am able to detect the $(x,y)$ coordinates of all markers for the transformed image and need to calculate the transformed $p$:
$tp = f(om1, om2, om3, op, tm1, tm2, tm3)$ where


*

*$om1, om2, om3$ are the original markers.

*$op$ is the original point of interest.

*$tm1, tm2, tm3$ are the transformed markers.

*$tp$ is the transformed point of interest.


There are two distinct questions here:


*

*How can I calculate coordinates for the transformed $(p)$?

*What corner cases should be factored in?


 
 A: As clarified in the comments, the transforms considered amount to all similarity transforms of the plane. Any such transform can be written as a product $TRS$, where $T$ is a translation, $R$ is a rotation and $S$ is a homogeneous scaling. In coordinates, this is
$$
\pmatrix{x'\\y'}=\pmatrix{\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha}\pmatrix{s&0\\0&s}\pmatrix{x\\y}+\pmatrix{t_x\\t_y}\;.
$$
I'll assume that the aspect ratio of the image is such that the rotations up to $15^\circ$ can't tilt the diagonal beyond the horizontal or vertical, that is, the markers on the top stay on the top and the markers on the right stay on the right. If not, you may have to do some additional tests first to identify the markers correctly.
The rotation angle $\alpha$ can be extracted from the top markers: $\alpha=\arctan((y_2'-y_1')/(x_2'-x_1'))$, where $(x_1',y_1')$ are the coordinates of the top left marker and $(x_2',y_2')$ are the coordinates of the top right marker. (This assumes that the markers originally have the same $y$ coordinate; if not, you need to subtract the corresponding angle calculated from the original coordinates.)
The scaling factor $s$ can be determined from the change in the distance between any two markers, say the top markers: $s=\sqrt{(x_2'-x_1')^2+(y_2'-y_1')^2}/(x_2-x_1)$, where $x_1$ and $x_2$ are the original $x$ coordinates of the top markers. (Again this assumes that their original $y$ coordinates are equal; if not, the denominator is a square root analogous to the numerator.)
Now the translation $(t_x,t_y)$ can be determined from any marker, say the top left marker:
$$
\pmatrix{t_x\\t_y}=\pmatrix{\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha}\pmatrix{s&0\\0&s}\pmatrix{x_1\\y_1}+\pmatrix{x_1'\\y_1'}\;.
$$
Then you just have to apply the transformation equation to $p$ to transform it.
