Closest 3x3 rotation matrix where all entries are in {1, 0, -1} Let $M$ be a 3x3 right-handed rotation matrix.
I need to find a closed form solution for the right-handed rotation matrix $M'$ where its entries are lie in ${1, 0, -1}$.
An initial idea is to replace each row of $M$ by the unit norm vector that is collinear to its largest component; i.e if the first row is $[0.126, -0.946, -0.299]$, then it should be replaced by $[0, -1, 0]$.
Any thoughts on this?
Is there a better approach?
What if one of the rotation angles is 45 degrees and the max component is not unique?
 A: To put the problem in its context, this is needed to extract sagittal/axial/coronal slices from MR and CT volumes. The volume files come with rotation matrices that express how the volume is oriented with respect to the standard basis which is the scanner's.
Sometimes the slices aren't taken from an orthogonal direction but there's a slight displacement in rotation, so I need to find the closest orthogonal rotation (swap of axis and directions) to know what orthogonal view the volume is taken from.
I tried and implemented my solution where I replace each row of my matrix with the unit norm vector on its largest component in absolute value and it works, all my volumes now have an orthogonal rather than oblique rotation.
A: The columns (or equivalently rows) of any orthogonal matrix are non-zero orthogonal vectors and it is quite easy to take 3 orthogonal vectors with only 1, -1 or 0 as entries. "Closest" is a very vague notion until you specialise some norm on the vector space of 3x3 matrices.
Looking at the original motivation I do not fully understand OP's own answer (what is an orthogonal rotation? For me any rotation is represented by an orthogonal matrix), but maybe this helps: say that the reference frame of the scanner is identified with the canonical basis $\{e_i\}$ of $\mathbb{R}^3$, that is $e_1= (1,0,0), \ldots$. Then, if I understand what the OP wrote, by the usual facts about the matrix associated to a linear map, the $i$-th column of your rotation matrix $R$ is the vector $R(e_i)$, that is the rotated basis vector (more precisely the column gives you the components of $R(e_i)$ with respect to the basis $\{e_i\}$).
