By using properties of determinants show that determinant is equal to $(1+a^2+b^2)^3$ $$\begin{vmatrix}1+a^2-b^2&2ab&-2b\\
2ab&1-a^2+b^2&2a\\
2b&-2a&1-a^2-b^2\end{vmatrix}=(1+a^2+b^2)^3$$
I have been trying to solve the above determinant. But unfortunately my answer is always coming as:
$$1+3a^2+3a^4+a^6+3a^2b^4+3b^2+4a^2b^2+a^4b^2+b^6+3b^4$$
 Please help me to solve this problem.
 A: $$\begin{vmatrix}1+a^2-b^2&2ab&-2b\\
2ab&1-a^2+b^2&2a\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$
$$=\begin{vmatrix}1+a^2-b^2&2ab&-2b\\
2ab-a(2b)&1-a^2+b^2-a(-2a)&2a-a(1-a^2-b^2)\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$  (applying $R'_2=R_2-aR_3$)
$$=\begin{vmatrix}1+a^2-b^2&2ab&-2b\\
0&1+a^2+b^2&a(1+a^2+b^2)\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$
$$=(1+a^2+b^2)\begin{vmatrix}1+a^2-b^2&2ab&-2b\\
0&1&a\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$
$$=(1+a^2+b^2)\begin{vmatrix}1+a^2-b^2+b(2b)&2ab+b(-2a)&-2b+b(1-a^2+b^2)\\
0&1&a\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$  (applying $R'_1=R_1+bR_3$)
$$=(1+a^2+b^2)\begin{vmatrix}1+a^2+b^2&0&-b(1+a^2+b^2)\\
0&1&a\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$
$$=(1+a^2+b^2)^2\begin{vmatrix}1&0&-b\\
0&1&a\\
2b&-2a&1-a^2-b^2\end{vmatrix}$$
$=(1+a^2+b^2)^2$ $\begin{vmatrix}1&0&-b\\
0&1&a\\2b+2a(0)-2b(1)&-2a+2a(1)-2b(0)&1-a^2-b^2+2a(a)-2b(-b)\end{vmatrix}$
(applying $R'_3=R_3+2aR_2-2bR_1$)
$=(1+a^2+b^2)^2$ $\begin{vmatrix}1&0&-b\\0&1&a\\0&0&1+a^2+b^2\end{vmatrix}$
$=(1+a^2+b^2)^3$
A: Assuming $a,b\in\mathbb{R}$ (and if not, proving it for real $a,b$ should allow you it to extend it to complex $a,b$), each row vector has the norm $(1+a^2+b^2)$ and the row vectors are all orthogonal so the matrix divided by $(1+a^2+b^2)$ is an orthogonal matrix and the absolute value of its determinant is 1 which immediately implies your result.
