# My calulations show that for all $n\times m$ matrices $A, B$, it is $det(A^tB)=0$. Where is the mistake?

Let $$A, B \in \mathbb{M}^{n\times m}(\mathbb{R})$$ be two $$n\times m$$ real matrices. Then the product $$A^t\cdot B$$ is a $$m\times m$$ real matrix, thus, it has a determinant $$D=det(A^t B)$$.

I calculated $$D$$ using the Leibniz Formula and my calulations result in $$D=0$$. I know this is wrong: for example if $$A=B$$ a square invertible matrix, then $$D=det(A^tA)= det(A^t)det(A)=\big(det(A)\big)^2\neq 0$$

So, unavoidably, there is a mistake in my calulations, but I cannot find it. I would apreciate any hints, corrections, or any insight. I would deeply apreciate if you could show me the correct calculations.

First, let $$A=(a_{i,j})_{i,j}$$, hence $$A^t=(a_{j,i})_{i,j}$$ and $$B=(b_{i,j})_{i,j}$$ for $$i=1,...,n$$ and $$j=1,...,m$$. We calculate the $$(i,j)$$-th term of the matrix $$A^t B$$ to be: $$(A^t B)_{ij} = \sum_{k=1}^n a_{k,i}b_{k,j}$$

Now we calculate the determinant of $$A^t B$$: $$\begin{array}{rlc} D &= \displaystyle \sum_{\sigma \in S(m)} \left(sgn(\sigma)\prod_{i=1}^m (A^t B)_{i, \sigma(i)}\right) & \text{(Leibniz formula for determinant)}\\ &= \displaystyle \sum_{\sigma \in S(m)} \left( sgn(\sigma)\prod_{i=1}^m \sum_{k=1}^n a_{k,i}b_{k,\sigma(i)} \right) & \text{(substitution)}\\ &= \displaystyle \sum_{\sigma \in S(m)} \left( sgn(\sigma)\sum_{k_1=1}^n\sum_{k_2=1}^n...\sum_{k_m=1}^n a_{k_1,1}a_{k_2,2}...a_{k_m,m}b_{k_1,\sigma(1)}b_{k_2,\sigma(2)}...b_{k_m,\sigma(m)} \right) & \text{(distributive property)}\\ & =\displaystyle \sum_{\sigma \in S(m)} \left( sgn(\sigma)\sum_{k_1=1}^n...\sum_{k_m=1}^n a_{k_1,1}...a_{k_m,m}b_{k_1,1}b_{k_2,2}...b_{k_m,m}\right) & (\sigma\text{ is a bijection)}\\ & = \displaystyle \left( \sum_{k_1=1}^n...\sum_{k_m=1}^n a_{k_1,1}...a_{k_m,m}b_{k_1,1}b_{k_2,2}...b_{k_m,m} \right) \left( \sum_{\sigma \in S(m)} sgn(\sigma) \right) & \text{(distributive property)}\\ & = 0 & \left( \sum_{\sigma \in S(m)} sgn(\sigma) = 0 \right) \end{array}$$

In the step where you write all $$b_{k_i,\sigma(i)}$$ to be $$b_{k_i,i}$$ in the product inside the summation stating the reason to be that $$\sigma$$ is a bijection, you're actually imposing $$\sigma=\textrm{id}$$.

Let us reduce ourselves to say $$n=3$$; then the body inside the summation is $$a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,\sigma(1)}b_{k_2,\sigma(2)}b_{k_3,\sigma(3)}$$, your argument reduces this to $$a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,1}b_{k_2,2}b_{k_3,3}$$ for all $$\sigma\in S_3$$ which disregards all the non-identity permutations.

For example, for say $$\sigma=(1 2)\in S_3$$, the corresponding term in the sum should be $$a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,2}b_{k_2,1}b_{k_3,3}$$ while the term in your argument stays the same $$a_{k_1,1}a_{k_2,2}a_{k_3,3}b_{k_1,1}b_{k_2,2}b_{k_3,3}$$ which is not necessarily the same.

One good way to find such mistakes is to take some particular example and calculate numbers on every step.

Lets take $$A = B = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$$.

On $$3$$rd line you have (given that only terms with $$k_1 = 1$$, $$k_2 = 2$$ are non-zero) $$\operatorname{sgn}((1)(2)) \cdot (a_{11} a_{22} b_{11}b_{22}) + \operatorname{sgn}((12))\cdot(a_{11}a_{22}b_{12}b_{21}) = 1$$, as it should be.

However, on the 4th line you have $$\operatorname{sgn((1)(2))}\cdot (a_{11}a_{22}b_{11}b_{22}) + \operatorname{sgn((12))}\cdot (a_{11}a_{22}b_{11}b_{22}) = 0$$. So the mistake is in this transition.

The reason is that, while $$\sigma$$ is a bijection and you have the same terms if you look on $$b$$ only, different $$a$$-s are attached to different $$b$$-s depending on permutation.