Trying to find $\det (B)$ Let $A=\left(\begin{matrix}1&1&-1\\-1&1&1\\1&-1&1\end{matrix} \right)$,and ${A}^{T}B{\left( \cfrac{1}{2}{A}^{T}\right)}^{T}-8{A}^{-1}B=I$, How to compute $\left|B \right|$?。
 A: Consider the permutation matrix $$P = \pmatrix{0 & 1 & 0\cr 0 & 0 & 1\cr 1 & 0 & 0\cr}$$
Note that $P^3 = I$ and $A = I + P - P^2$.  Since $A P^2 = P^2 + I - P = -A + 2 I$, we see
that $A^{-1} = (P^2 + I)/2$, while $A^T = I + P^2 - P$.  It seems reasonable, then, that a solution $B$ might also be a linear combination of $I$, $P$ and $P^2$.  In fact, if 
$B = a I + b P + c P^2$ we get 
$$ \eqalign{\frac{1}{2} A^T B A &- 8 A^{-1} B = \frac{1}{2} (I + P^2 - P)(a I + b P + c P^2)(I + P - P^2) - 4 (P^2 + I) (aI + b P + cP^2)\cr
&= \frac{-5 a - 9 b-c}{2} I + \frac{-a-5b-9c}{2} P + \frac{-9 a - b - 5 c}{2} P^2}  $$
For this to be $I$, we just have to solve the $3 \times 3$ system
$$ \eqalign{ 5a + 9 b + c &= -2\cr
             a + 5 b + 9 c &= 0\cr
             9 a + b + 5 c &= 0\cr}$$
obtaining $a = -2/45, b = -19/90, c = 11/90$.
Now note that the eigenvalues of $P$ are the three cube roots of $1$, say $1, \omega, \omega^2$; the eigenvalues of $B = a I + b P + c P^2$ are then $a+b+c$, $a+b\omega + c \omega^2$ and $a + b\omega^2 + c \omega$, so (using $1 + \omega + \omega^2 = 0$)
$$\eqalign{\det(B) &= (a+b+c)(a+b\omega + c \omega^2)(a + b \omega^2 + c \omega)\cr
     &= a^3 + b^3 + c^3 - 3 a b c\cr} $$ 
Substituting the values of $a,b,c$ gives us the answer $-1/90$.
EDIT: Note that this shows that there is a $B$ with determinant $-1/90$, but not that this is the only solution.
The calculation can be shortened: if $B$ is of the form $aI + bP + cP^2$, then it commutes with $A$.  So we can simplify the equation to
$$ \left(\frac{A^T A}{2} - 8 A^{-1}\right) B = I$$
Thus $B = \left(\dfrac{A^T A}{2} - 8 A^{-1}\right)^{-1}$, and 
$\det B = 1/\det  \left(\dfrac{A^T A}{2} - 8 A^{-1}\right)$.
We just have to calculate $\det  \left(\dfrac{A^T A}{2} - 8 A^{-1}\right) = -90$ to say $\det(B)=-1/90$.
A: Edit. Revisiting this question, I wonder why I couldn't solve for $B$ directly by paper and pencil in my previous answer. The equation in the OP can be rewritten as
$$
\frac12A^TBA - 8A^{-1}B = I.\tag{1}
$$
We first show that it has a unique solution. Consider the homogeneous linear equation $\frac12A^TBA - 8A^{-1}B = 0$. Let $J$ be the all-one matrix. Then $AA^T=4I-J$. Therefore
\begin{align}
\frac12A^TBA - 8A^{-1}B = 0
&\Rightarrow AA^TBA - 16B = 0\\
&\Rightarrow (4I-J)BA - 16B=0\\
&\Rightarrow 4B(A-4I) = JBA\tag{2}\\
&\Rightarrow 4B = JBA(A-4I)^{-1},\tag{3}
\end{align}
where $A-4I$ has an inverse because it is strictly diagonally dominant. Therefore, $B=JX$ for some matrix $X$. It follows that $JB=J^2X=3B$. Hence $(2)$ gives $4B(A-4I) = 3BA$, or $B(A-16I)=0$. Again, $A-16I$ is strictly diagonally dominant and hence nonsingular. Thus the equation $\frac12A^TBA - 8A^{-1}B = 0$ possesses only the trivial solution and $(1)$ is uniquely solvable.
Note that $AA^T$ is also equal to $2A-A^2$. So, we may rewrite $(1)$ as
$$
(2A-A^2)BA - 16B = 2A.\tag{4}
$$
Therefore, it makes sense to try some $B$ that is a polynomial in $A$, but in this case, as indicated in the other answer, $A$ and $B$ commute. Therefore $(1)$ can be rewritten as $B=\left(\frac{A^T A}{2} - 8 A^{-1}\right)^{-1}$. As $\det\left(\frac{A^T A}{2} - 8 A^{-1}\right)=-90\ne0$, this $B$ exists and it is indeed the unique solution to $(1)$. Hence $\det(B)=\frac1{-90}$.
