What is the orthogonal complement Let $\langle A,B \rangle = \text{tr}(B^TA)$. For $M_n(\mathbb{R}$): Find the orthogonal complement of the diagonal-square-matrices.
So I need to find $$U^{\perp} = \left\{ A\in M_n(\mathbb{R}) \mid \forall B\in \{\text{diagonal-square matrices}\}: \langle A,B \rangle = 0 \right\}$$
Hence,
$$\langle A,B \rangle = 0 \iff \text{tr}(B^TA) = 0 \iff \text{tr}(BA) = 0$$
So $A$ is a square matrix and $B$ is every possible square-diagonal matrix.
So it must be that $A=0$, isn't it?
 A: It is not the case that $A$ must be $0$.  For example, when $n = 2$, the element
$$
A = \pmatrix{0&1\\0&0}
$$ 
is a member of $U^\perp$.
Hint: Because $B$ is diagonal, we have
$$
\operatorname{trace}(BA) = \sum_{i=1}^n  a_{ii}b_{ii}
$$
Where $a_{ij}$ and $b_{ij}$ are the entries of $A$ and $B$ respectively.  Try different combinations of $b_{ii}$.  Note that the above sum says nothing about $a_{ij}$ when $i \neq j$.
A: Think of this class of matrices as vectors. If 
$A = \{x \in \mathbb R^n \mid x_i = 0, \forall i\in I\}$ 
then 
$A^\perp = \{y \in \mathbb R^n \mid x^Ty = 0\} = \{y \in \mathbb R^n \mid y_i = 0, \forall i\not\in I\}$. 
(To convince yourself that it has to be for all $i\in I$, just use the indicator vectors for all $i\in I$ as a counterexample. All $e_i, i\in I$ are in $A$, and therefore every element in $B$ must be $0$ for all $i\in I$. But maybe that is trivial.)
In your case, $I$ is merely the set of indices corresponding to the off-diagonal entries.  So if 
$A = \{X \in \mathbb R^{n\times n} \mid X_{ij} = 0 \forall i\neq j\}$ 
then 
$A^\perp = \{X \in \mathbb R^{n\times n} \mid X_{ij} = 0 \forall i=j\}$. 
Note that this set is much bigger than $\{0\}$.
