Maximal dimension of subspace of matrices whose products have zero trace

Question

In the space of all real matrices with dimension $n$, find the maximal possible dimension of a subspace $V$ such that $\forall X,Y\in V,\, \operatorname{tr}(XY)=0$.

Answer 1

If $V$ is a subspace of real matrices of size $n$ with the property that $\operatorname{tr}(XY)=0$ for all $X,Y\in V$, then $\operatorname{tr}(X^2)=0$ for all $X\in V$.

Let $W$ be a subspace of real matrices of size $n$ with the property that $\operatorname{tr}(X^2)=0$ for all $X\in W$. Then $\operatorname{tr}((X+Y)^2)=0$ for all $X,Y\in W$. But $\operatorname{tr}((X+Y)^2)=\operatorname{tr}(X^2)+\operatorname{tr}(Y^2)+2\operatorname{tr}(XY)$. It follows that $\operatorname{tr}(XY)=0$ for all $X,Y\in W$.

This shows that the problem reduces to the following: Investigations about the trace form which is already solved.

Answer 2

(This is not a difficult problem, but it is also not an easy homework problem. I wonder if the original problem is much simpler and the OP has copied the problem statement wrongly.)

Note that $\operatorname{tr}(XY)=\operatorname{vec}(Y^T)^T\operatorname{vec}(X)= \operatorname{vec}(Y)^TP\operatorname{vec}(X)$, where $P$ is the symmetric permutation matrix defined by $P_{(j-1)n+i,\ (i-1)n+j}=P_{(i-1)n+j,\ (j-1)n+i}=1$ for $i,j\in\{1,\ldots,n\}$ and other entries zero. Since the matrix $P$ is orthogonally similar to $I_p\oplus (-I_q)$, where $q=n(n-1)/2$ and $p=n^2-q$, a maximal matrix subspace $V$ such that $\operatorname{tr}(XY)=0$ whenever $X,Y\in V$ can be identified with a maximal subspace $W\subset\mathbb{R}^{n^2}$ such that $u^T(I_p\oplus(-I_q))v=0$ whenever $u,v\in W$.

We now claim that $\dim W\le q$. Suppose the contrary. Then there exists a set of $q+1$ linearly independent vectors $u_1,\ldots,u_{q+1}\in\mathbb{R}^{n^2}$ such that $u_i^T(I_p\oplus(-I_q))u_j=0$ for all $i,j=1,\ldots,q+1$. Hence there exists a nonzero linear combination $x=\sum_i c_iu_i\not=0$ such that the last $q$ entries of $x$ are zero. Write $x^T=(\tilde{x}^T,0_{1\times q})$. Then $x^T(I_p\oplus(-I_q))x=\|\tilde{x}\|^2\not=0$ because $\tilde{x}\not=0$, but we also have $x^T(I_p\oplus(-I_q))x=0$ because $x$ is a linear combination of $u_1,\ldots,u_{q+1}$. So we arrive at a contradiction. Therefore $\dim W\le q$.

Finally, the dimension $q$ is attainable. For example, consider the set of all strictly upper triangular matrices. Hence the required maximal dimension is $q=n(n-1)/2$.