Checking if a linear transformation contains a shear I am using this approach, from a previous question when trying to extract the transformation matrix from one set of points to another. The main difference is that I use homogenous coordinates. For example, consider I have the following set of points: $(1,1), (1,3), (2,1)$ and $(1,1), (5,1), (5,3)$
Hence, I have
$$
X=
\begin{pmatrix}
1 & 1 & 1\\
1 & 3 & 1\\
2 & 1 & 1
\end{pmatrix}
$$
and
$$
X'=
\begin{pmatrix}
1 & 1 & 1\\
5 & 1 & 1\\
5 & 3 & 1
\end{pmatrix}.
$$
When I put this into the equation from the link I obtain my transformation matrix $T$ as:
$$
T=
\begin{pmatrix}
4 & 2 & 0\\
2 & 0 & 0\\
-5 & -1 & 1
\end{pmatrix}
$$
My question is how can I check whether or not the transformation includes a shear?
EDIT
I can assume uniform scaling.
Based on the fact that I can assume uniform scaling and this question, is it a matter of checking if the length of the leftmost column vectors are the same? (after extracting the translation components?)
 A: Ignoring any translation, your transformation applies the unit vector $(\cos\theta,\sin\theta)$ to $(4\cos\theta+2\sin\theta,2\cos\theta)$. The latter does not have a constant length.
A: Yes, you function is skewed (shear) although I cannot pin-point it in your matrix.
Your original point points map onto a cartesian plane something like this:

Your transformed points look something like this:

If your points all follow a singular, same transformation for all of them, then they could be considered as a whole:

In the above image, the green triangle is being transformed to the red one. The triangle is in picture above proves that the triangular plane is being stretched as one of the points remains the same while the others change, meaning it has to sheared.
Another way to prove that your graph is sheared:
You can see that for $y = 1$, it is transformed to $y = 1$ in your first point, but also to $y = 3$ in your third point. This is physically not possible with just one linear equation for both points at once.
Finding the shear transformation:
I do not know how to find it in 3x3 matrix but using a 2x2 matrix a function can be sheared by apply the following matrix transformation:
$$T \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 1 & m \\ k & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}$$
,where, $\begin{bmatrix} 1 & m \\ k & 1 \end{bmatrix}$ is the shear transformation ( m = horizontal shear, k = vertical shear).
From here, I suggest creating 3 simultaneous equations from the 3 points, as linear matrices can be expressed as system of equations, to solve for your shear variables, $m \text{ and } k$.
