# Dimension of nullspace of difference of two rigid transformations

Given distinct proper rigid transformations $A, B \in \operatorname{SE}(n)$, what is the maximum dimension of the nullspace of $A - B$? That is, what is the maximum dimension of $\operatorname{Eq}(A, B) = \{x \in \mathbb{R}^n\ |\ A(x) = B(x)\}$?

I conjecture that the answer is $n - 2$ by observing simple cases:

When $n = 2$, proper rigid transformations are "defined" by $2$ affinely independent points, so $\operatorname{Eq}(A, B)$ can contain at most one point and has a dimension of $0$.

When $n = 3$, proper rigid transformations are "defined" by $3$ affinely independent points, so $\operatorname{Eq}(A, B)$ is at most the affine hull of two affinely independent points and has a dimension of $1$.

As this line of reasoning is not quite rigorous, I cannot tell if it easily extends to higher dimensions (also hampered by my lack of four-dimensional imagination!). Is my conjecture true for all integers $n \geq 2$?

-

Yes. WLOG, suppose $A$ and $B$ are linear (so there are no translations). Let $A,B$ also denote the matrix representations of the two transofrmations. As they belong to $SE(n)$, $A^TB\in SO(n)$. Hence $A^TB=UDU^\ast$ for some unitary matrix $U$ and some complex diagonal matrix $D$ whose diagonal entries have unit moduli. Therefore $$\operatorname{nullity}(A-B)=\operatorname{nullity}(I-D).$$ If $\operatorname{nullity}(A-B)=n-1$, $D$ must have $n-1$ eigenvalues equal to $1$. However, as $A^TB$ is real orthogonal and $A\not=B$, the remaining eigenvalue of $D$ must be equal to $-1$. But then $\det A^TB=-1$, which is a contradiction. Therefore the nullity of $A-B$ is at most $n-2$.
@HerngYi Suppose $A(x)=Px+p$ and $B(x)=Qx+q$, where $P,Q$ are matrices and $p,q$ are translation vectors. The nullspace of $A-B$ is then the set $H=\{x: (P-Q)x+(p-q)=0\}$. Suppose $x_0\in H$. Then $H=x_0+\{u: (P-Q)u=0\}$. So the dimension of the affine subspace $H$ is equal to the dimension of the linear subspace $\{u: (P-Q)u=0\}$. In other words, as long as the dimension of the nullspace is concerned, $\dim H$ can be calculated as if $p=q=0$. –  user1551 Feb 8 '13 at 9:08