# Product of upper triangular matrix and lower triangular matrix

I need to show the following:

Let $A,B \in GL(n)$ so that both of them are upper triangular matrices and $A*B^t$ or $A^t*B$ is a diagonal matrix. Show that $A$ and $B$ are diagonal matrices as well.

I tried it for small $n$ and it is kinda clear but i don't know how to prove it. I tried contraposition but it did not work well. Some help would be appreciated!

• This doesn't seem right as stated. Suppose $A=0$; then $AB^T=A^T B=0$ are both diagonal matrices, regardless of $B$. – Semiclassical May 25 '17 at 19:20
• You are implying 0 is $\in GL(n)$, which is false – Johny Hunter May 25 '17 at 19:28
• That'd do it. Objection withdrawn. – Semiclassical May 25 '17 at 19:33

I will assume that $*$ stands for matrix multiplication.
If $A.B^t$ is a diagonal matrix $D$, then $B^t=A^{-1}.D$. But both $A^{-1}$ and $D$ are upper triangular and therefore their product is upper triangular too. In other words, $B^t$ is upper triangular. But $B^t$ is lower triangular. So, $B$ is diagonal, and so $A(=B^{-1}.D)$ is diagonal too.