How to get a 2x2 transformation matrix for 2 triangles in 3D space. Lets say I have a triangle $v_{ij}$ in $R^3$. This triangle undergoes some deformation.
From what I understand I should be able to describe said triangle as 2 vectors, $v_i$, and $v_j$. I don't care about the location of the triangle, only its deformation. So if I were to define i and j as the basis vectors for a plane in $R^2$, then I should be able to describe said transformation as a $2x2$ matrix by flattening out the post deformation triangle $v_{îĵ}$ onto that same plane by basically pinning the two $v$ coordinates together, turning $î$ so that it matches the direction of $i$, and then rotating the entire triangle around $î$ such that $ĵ$ lies in the plane defined by $v_{ij}$.
However, I wasn't sure how to do this in a practical sense.
Could someone explain how, if I have vectors $[i_x,i_y,i_z], [j_x, j_y, j_z] \text{and} \ [î_x, î_y, î_z], [ĵ_x, ĵ_y, ĵ_z]$ I can get a $2x2$ matrix showing that deformation?
I am not a mathematician so I apologize for the lack of proper notation. All I know is that in the resulting matrix $ĵ_y$ should be 0.
 A: Consider a 2D triangle, with a vertex at origin $(0,0)$, another at $(A,0)$, and the third at $(B, C)$.  If we ignore translation and rotation, we can describe every possible triangle using just these three scalars $A$, $B$, and $C$.
To identify specific points within the triangle, instead of Cartesian 2D coordinates, we can use barycentric coordinates $(u, v, w)$.  Barycentric coordinates have the property that $u + v + w = 1$, so any pair of them suffices. From now on, I shall only include the two barycentric coordinates $u$ and $v$, as the third one is always $w = 1 - u - v$.
Points that are in the "same" location with respect to their triangles have the same barycentric coordinates, even when the triangles differ.  For example, the barycenter of the triangle is always at coordinates $\left(\frac{1}{3}, \frac{1}{3}\right)$. (Note that when the third barycentric coordinate is included, $w = 1 - u - v$, we have $u = v = w = \frac{1}{3}$ at the barycenter.)
If we choose the Cartesian coordinate system origin to have barycentric coordinates $(0, 0)$, the first vertex $(1, 0)$ (at Cartesian coordinates $(A, 0)$), and the second vertex $(0, 1)$ (at Cartesian coordinates $(B, C)$), the conversion between coordinate systems is
$$\left\lbrace \begin{aligned}
x &= u A + v B \\
y &= v C \\
\end{aligned} \right . \quad \iff \quad \left\lbrace \begin{aligned}
u &= \frac{x C - y B}{A C} \\
v &= \frac{y}{C} \\
\end{aligned} \right . \tag{1}\label{G1}$$
Let's say we have a triangle defined by $A_1$, $B_1$, and $C_1$, and a point inside it at Cartesian coordinates $(x_1, y_1)$.  By applying $\eqref{G1}$, we can check if the triangle is changed into $A_2$, $B_2$, and $C_2$, and what the Cartesian coordinates become $(x_2, y_2)$, if the two points' barycentric coordinates remain unchanged:
$$\left\lbrace\begin{aligned}
x_2 &= x_1 \frac{A_2}{A_1} + y_1 \frac{A_1 B_2 - A_2 B_1}{A_1 C_1} \\
y_2 &= y_1 \frac{C_2}{C_1} \\
\end{aligned} \right. \tag{2a}\label{G2a}$$
We can describe this using a 2×2 matrix $\mathbf{T}_C$,
$$\mathbf{T}_C = \left[ \begin{matrix}
\frac{A_2}{A_1} & \frac{A_1 B_2 - A_2 B_1}{A_1 C_1} \\
0 & \frac{C_2}{C_1} \end{matrix} \right] \tag{2b}\label{G2b}$$
Note the diagonal elements are the scale factors along Cartesian $x$ and $y$ axes, and the upper right term is the only cross term. When the cross term is nonzero, the points get "skewed".
Similarly, we can examine how the barycentric coordinates differ when the Cartesian coordinates stay the same:
$$\left\lbrace\begin{aligned}
u_2 &= \frac{A_1}{A_2} u_1 + \frac{B_1 C_2 - B_2 C_1}{A_2 C_2} v_1 \\
v_2 &= \frac{C_1}{C_2} v_1 \\
\end{aligned} \right. \tag{3a}\label{G3a}$$
which in matrix form is $\mathbf{T}_B$,
$$\mathbf{T}_B = \left[ \begin{matrix}
\frac{A_1}{A_2} & \frac{B_1 C_2 - B_2 C_1}{A_2 C_2} \\
0 & \frac{C_1}{C_2} \\
\end{matrix} \right] \tag{3b}\label{G3b}$$
Note: $\mathbf{T}_C$ applies to 2D Cartesian coordinates $(x, y)$, whereas $\mathbf{T}_B$ applies to barycentric coordinates $(u, v)$.

In three dimensions, given a triangle having vertices $\vec{p}_0$, $\vec{p}_0 + \vec{p}_u$, and $\vec{p}_0 + \vec{p}_v$, the barycentric coordinates $(u, v)$ correspond to point $\vec{p}$,
$$\vec{p} = \vec{p}_0 + u \vec{p}_u + \vec{p}_v \tag{4a}\label{G4a}$$
If we use $\vec{p} = (x, y, z)$, $\vec{p}_0 = (x_0, y_0, z_0)$, $\vec{p}_u = (x_u, y_u, z_u)$, and $\vec{p}_v = (x_v, y_v, z_v)$, then $\eqref{G4a}$ in Cartesian coordinates is
$$\left\lbrace \begin{aligned}
x &= x_0 + u x_u + v x_v \\
y &= y_0 + u y_u + v y_v \\
z &= z_0 + u z_u + z z_v \\
\end{aligned} \right. \tag{4b}\label{G4b}$$
The inverse is slightly more complicated, because we have only two unknowns ($u$ and $v$) but three equations. Depending on which pair of equations we use to solve for $u$ and $v$, we have three solutions:
$$\left\lbrace\begin{aligned}
u &= \frac{(x - x_0) y_v - (y - y_0) x_v}{x_u y_v - x_v y_u} \\
v &= \frac{x_u (y - y_0)  y_u (x - x_0)}{x_u y_v - x_v y_u} \\
\end{aligned} \right. \tag{4c}\label{G4c}$$
$$\left\lbrace\begin{aligned}
u &= \frac{(x - x_0) z_v - x_v (z - z_0)}{x_u z_v - x_v z_u} \\
v &= \frac{x_u (z - z_0) - (x - x_0) z_u}{x_u z_v - x_v z_u} \\
\end{aligned} \right. \tag{4d}\label{G4d}$$
$$\left\lbrace\begin{aligned}
u &= \frac{(y - y_0) z_v - y_v (z - z_0)}{y_u z_v - y_v z_u} \\
v &= \frac{y_u (z - z_0) - (y - y_0) z_u}{y_u z_v - y_v z_u} \\
\end{aligned} \right. \tag{4e}\label{G4e}$$
For numerical stability, it is a good idea to calculate $x_u y_v - x_v y_u $, $x_u z_v - x_v z_u$, and $y_u z_v - y_v z_u$, and use the equation above that has the largest divisor in magnitude (i.e., largest absolute value).
A: Compute the lengths $\|v_i\|, \|v_j\|$ and the angle between the two vectors $\cos\theta_{ij}=\dfrac{v_i\cdot v_j}{\|v_i\| \|v_j\|}$. From this, you draw the triangle in a 2D reference frame. The points are $(0,\|v_i\|)$ and $(\|v_j\|\cos\theta_{ij},\|v_j\|\sin\theta_{ij})$. [You can also compute this without trigonometric functions.]
Repeat for the deformed triangle.
Now you solve
$$\begin{pmatrix}0&x'_1\\y'_0&y'_1\end{pmatrix}=M\begin{pmatrix}0&x_1\\y_0&y_1\end{pmatrix},$$ or
$$M=\begin{pmatrix}0&x'_1\\y'_0&y'_1\end{pmatrix}\begin{pmatrix}0&x_1\\y_0&y_1\end{pmatrix}^{-1}.$$
