When reducing $A$ to $I$ via elementary matrices $E_k...E_1A=I$, why is it true also that $AE_k...E_1=I$

Question

Say we have a $3 \times 3$ matrix $A$ and we perform the row operation $Z_1:(R_1 \iff R_2)$ of switching the first and second row to get $A'$.

Now, if we perform the same row operation $Z_1:(R_1 \iff R_2)$ on the identity matrix $I_3$ we get the elementary matrix $E_1$ and it is a common fact of linear algebra that $E_1A = A'$. Note that it is important that we multiply $E_1$ on the left of $A$.

Now, say further that $A$ is invertible and that we want to find $A^{-1}$. Then what we do is set up the augmented matrix $[A | I_3]$ and perform row operations $Z_1,...,Z_k$ until we we get an equivalent augmented matrix of the form $[I_3|B]$. It is a common fact of linear algebra that $B = A^{-1}$.

Now, the row operations $Z_1,...,Z_k$ correspond to the elementary matrcies $E_1,...,E_k$ and we can thus see that $A^{-1}=E_k...E_1$.

This is great and makes perfect sense! We reduced $A$ to $I_3$ via row operations corresponding to elementary matrices, so it is clear that $A^{-1}A = E_k,...E_1A=I_3$.

However, it is also true that $AA^{-1}=I_3$ and thus we have that $AE_k,...E_1 = I_3$. In the case when applying a string of elementary matrices to a matrix $A$ such that the string reduces $A$ to the identity, it is no longer important whether we apply the multiplication of the matrices on the right or left.

The bold font was meant to emphasize the phenomonon that I was hoping somebody could clarify. Understand that I am not asking why $A^{-1}A = AA^{-1}$ but rather seeking to clarify the more subtle observation of how the elementary matrices that embody the row operations can be applied to either side of $A$ in the case when they reduce $A$ to $I$.

Thank you.

Answer 1

I'm going to work with a $2\times 2$ example so that it is explicit and clear in each step what is happening. Let $A = \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix}$, then

$$ \begin{pmatrix} \frac{1}{4} & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & -5 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & \frac{1}{11} \end{pmatrix} \begin{pmatrix} 1 & 0 \\ \frac{1}{2} & 1 \end{pmatrix} \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \tag{1}$$

Multiplying on the left

Step by step (so we can compare with multiplying on the right shortly):

$$ \begin{pmatrix} 1 & 0 \\ \frac{1}{2} & 1 \end{pmatrix} \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} = \begin{pmatrix} 4 & 5 \\ 0 & \frac{11}{2} \end{pmatrix} \tag{2}$$

Rescaling,

$$ \begin{pmatrix} 1 & 0 \\ 0 & \frac{2}{11} \end{pmatrix} \begin{pmatrix} 1 & 0 \\ \frac{1}{2} & 1 \end{pmatrix} \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} = \begin{pmatrix} 4 & 5 \\ 0 & 1 \end{pmatrix} \tag{3}$$

Then

$$ \begin{pmatrix} 1 & -5 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & \frac{2}{11} \end{pmatrix} \begin{pmatrix} 1 & 0 \\ \frac{1}{2} & 1 \end{pmatrix} \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} = \begin{pmatrix} 4 & 0 \\ 0 & 1 \end{pmatrix} \tag{4}$$

and finally

$$ \begin{pmatrix} \frac{1}{4} & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & -5 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & \frac{2}{11} \end{pmatrix} \begin{pmatrix} 1 & 0 \\ \frac{1}{2} & 1 \end{pmatrix} \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \tag{5}$$

Let's then work this the opposite way to see what the row operations above are doing when we multiply on the right instead.

Multiplying on the right

$$ \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} \begin{pmatrix} \frac{1}{4} & 0 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 5 \\ -\frac{1}{2} & 3 \end{pmatrix} \tag{6}$$

So in the first part, this gave us a unit pivot in the upper left and it does the same here (which makes sense). Note that it actually became a column operation on the first column though as opposed to a row operation on the first row when multiplied on the left. Of course that distinction doesn't matter when multiplying diagonal matrices as in $(5)$..

$$ \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} \begin{pmatrix} \frac{1}{4} & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & -5 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ -\frac{1}{2} & \frac{11}{2} \end{pmatrix} \tag{7}$$

So our matrix $E_3$ again eliminated the upper right element.

$$ \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} \begin{pmatrix} \frac{1}{4} & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & -5 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & \frac{2}{11} \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ -\frac{1}{2} & 1 \end{pmatrix} \tag{8}$$

This one again was a column operation and instead of scaling the bottom row, it scaled the rightmost column. Finally,

$$ \begin{pmatrix} 4 & 5 \\ -2 & 3 \end{pmatrix} \begin{pmatrix} \frac{1}{4} & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & -5 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & \frac{2}{11} \end{pmatrix} \begin{pmatrix} 1 & 0 \\ \frac{1}{2} & 1 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \tag{9}$$

So again $E_1$ eliminated the bottom left element.

---

As we saw above, our rescaling elementary matrices keep that behavior, it's just a matter of whether it's a row or a column rescaling depending on if it is multiplied on the left or on the right. And you can see easily that if you had to switch rows, the same logic would apply. So the question then is: what are the elimination elementary matrices doing?

Let's look at equations $(4)$ and $(7)$ more closely. In $(3)$, our previous operations had made our matrix into upper triangular form and we were starting to eliminate upwards and so we had a unit pivot in the bottom right before eliminating upwards. In step $(6)$, we rescaled the upper left element. However in step $(7)$, we didn't get a $0$ below, but rather to the right. So what that means is that the elimination matrices do not eliminate upwards when multiplied on the right but rather eliminate to the right. Similarly, eliminating downwards is actually the same as eliminating to the left when multiplying on the right which you can see when you compare equation $(2)$ to equation $(9)$. The pivot that was created in the bottom right was then used to eliminate the element to the left of it. (To know that it's right/left, pay close attention to what the matrix multiplication is doing and which elements are at play.)

---

So in short: row rescalings get turned into column rescalings, row swaps get turned into column swaps, and eliminating downward (upward) gets turned into eliminating leftward (rightward).

Answer 2

If $$E_2E_1A=I\tag{1},$$ then $A=E_1^{-1}E_2^{-1}$ and thus $$ AE_2E_1=(E_1^{-1}E_2^{-1})E_2E_1=I\tag{2} $$

This simply says that if you can transform $A$ into the identity matrix by pure row operations, you can also do it by pure column operations.

But the orders of the operations you do are different.

In (1), you first do the row operation $E_1$ and then the operation $E_2$. But in (2), you first do the column operation $E_2$ and then the operation $E_1$.

it is no longer important whether we apply the multiplication of the matrices on the right or left

If this were true, what you should expect is $$ E_2E_1A=AE_1E_2 $$ which is in general not true.