$A$ and $B$ are positive self-adjoint matrix such that $AB$ is self-adjoint then $AB$ is positive Question: $A$ and $B$ are positive self-adjoint matrix such that $AB$ is self-adjoint then prove that $AB$ is positive.
My first try: Here I can see that the same question was there but it has a little amount of typo. What I have to prove is that $(x, ABx)>0$ for all $x>0$.
Now, $(x,ABx)=(Ax,Bx)=(BAx,x)$ now we also have that $AB=BA$ because $AB=(AB)^*=B^*A^*=BA$ because $A,B,AB$ are all self-adjoint and we can also see that $(BAx,Ax)>0$ but I can't get rid of $A$ in the 2nd co-ordinate.
My second try: Then I tried using this result that "if $A$ and $B$ are positive self-adjoint matrix then there exist a basis $x_1,\cdots, x_n$ of $X$ satisfies an eqn of the form $$Ax_j=\lambda_j Bx_j.$$
where $\lambda_j$ is real."
Now $A,B$ are positive $\Rightarrow \lambda_j>0$ using Generalised Raleigh quotient then $$(x_j,ABx_j)=(Ax_j,Bx_j)=(\lambda_j Bx_j,Bx_j)=\lambda_j||Bx_j||^2>0.$$
Hence $AB$ is positive.
Here I am confused that what is the place where I used $AB$ is self-adjoint! Please help. You can also help me with different proofs.
 A: Let us start with the positive operators $A$, $B$ as in the OP. Then $AB$ self-adjoint implies
$$
AB = (AB)^*=B^*A^*=BA\ ,
$$
so that $AB$ are commuting. Using continuous functional calculus w.r.t. the function $f(x)=\sqrt x$ defined on $[0,\infty)$ (and this domain contains the spectrum of $A$, and the spectrum of $B$) we obtain square roots $S=f(A)$, $T=f(B)$, i.e. $A=S^2$, and $T=B^2$.
Explicitly, for $A$ only, let $p_n$ be a sequence of polynomials such that $p_n\to f$ on a compact interval (for instance $[0,\|A\|]$) contaning the spectrum of $A$. Such a sequence is insured by Stone-Weierstraß. Then $$p_n(A)\to f(A)=:S\ .$$
Here, $f(A)$ is defined as $\lim p_n(A)$, the limit exists, Cauchy sequence. (Functional calculus of bounded operators shows this does not depend on $(p_n)$, but we do not need this.)
We denote by $S$ this value.
Then
$$
\begin{aligned}
S^2
&=(f(A))^2=(\lim p_n(A))^2=\lim (p_n(A))^2
=\lim p_n^2(A)
\\
&=(\lim p_n^2)(A)
=(\lim p_n)^2(A)
\\
&=f^2(A)=\operatorname{id}(A)=A\ .
\\[3mm]
SB &=(\lim p_n(A))B=\lim p_n(A)B\\
&=\lim B p_n(A)=B(\lim p_n(A))=BS\ .
\\[3mm]
(S^*x, y)
&=(x,Sy)=(x,(\lim p_n(A))y)=(x,\lim p_n(A)y)=\lim (x,p_n(A)y)\\
&=\lim (p_n(A)x,y)=(\lim p_n(A)x,y)=((\lim p_n(A))x,y)\\
&=(Sx,y)\ ,\qquad\text{ for all $x,y$ in the given Hilbert space.}
\end{aligned}
$$
(We have used $AB=BA$. These properties are basic properties of the functional calculus. Starting from $AB=BA$ we get to $f(A)B=Bf(A)$, so $SB=BS$. Similarly, starting with $SB=BS$ we obtain $Sf(B)=f(B)S$. So $ST=TS$ the two operators $S,T$ also commute.)
Let now $x$ be $\ne 0$. We have in a row:
$$
(ABx,x) =(SSTTx,x)=(TSSTx,x)=(STx,STx)=\|STx\|^2>0\ .
$$
(We have $(STx,STx)=\|STx\|^2\ge 0$, and in case of an equality, from $(STx,STx)=0$ we get first $Tx=0$, since $S>0$, then from $(Tx,Tx)=0$ also $x=0$, since $T>0$. Contradiction, since we started with an $x\ne 0$.)

Alternatively, we could have intoduced only $T$, and have the same argument with $SS$ replaced by $A$, e.g. $(ABx,x)=(ATTx,x)=(TATx,x)=(ATx,Tx)>0$ since $Tx\ne 0$ since $(Tx,Tx)=(TTx,x)=(Bx,x)>0$.

Note: In case of finitely dimensional spaces, things are simple. The two commuting self-adjoint operators can be diagonalized simultaneously w.r.t. some ortonormal basis, and if $A=\operatorname{diag}(a_1,\dots,a_n)>0$, then $S=\sqrt A:=
\operatorname{diag}(\sqrt{a_1},\dots,\sqrt{a_n})>0$ is the explicit square root of $A$ (which is positive), it is diagonal w.r.t. the same basis, et caetera.

Note: See also

*

*Functional calculus for bounded operators using continuous functions

*Functional calculus for bounded operators using Borel functions
A: We can use a nice trick here given that both $A$ and $B$ are positive. As $A$ is positive, this implies that $A$ has a unique positive square root, $A^{\frac{1}{2}}$, which being positive has an inverse $A^{-\frac{1}{2}}$. Thus,
$$AB = A^{\frac{1}{2}}A^{\frac{1}{2}}BA^{\frac{1}{2}}A^{-\frac{1}{2}}$$
So we can see that $A^{\frac{1}{2}}BA^{\frac{1}{2}}$ is similar to AB. Thus showing $A^{\frac{1}{2}}BA^{\frac{1}{2}}$ is positive implies that $AB$ is positive.
For self-adjoint: $\left(A^{\frac{1}{2}}BA^{\frac{1}{2}}\right)^*=\left(A^{\frac{1}{2}}\right)^*B^*\left(A^{\frac{1}{2}}\right)^* = A^{\frac{1}{2}}BA^{\frac{1}{2}}$ because both $A^{\frac{1}{2}}$ and $B$ are self-adjoint.
For positive: For any $v\in V\setminus \{0\}$ as $A^{\frac{1}{2}}$ is invertible we have that $A^{\frac{1}{2}}v \neq 0$ and thus $\langle A^{\frac{1}{2}}BA^{\frac{1}{2}}v,v\rangle = \langle BA^{\frac{1}{2}}v, A^{\frac{1}{2}}v\rangle>0$ as $A^{\frac{1}{2}}$ is self-adjoint, and $B$ is positive.
Therefore $A^{\frac{1}{2}}BA^{\frac{1}{2}}$ is positive implying that $AB$ is positive as desired. In general when proving results given a positive operator it may be easier to prove the result for a similar operator using the above trick.
