One can show that $\sin(x)/\sqrt{5/4-\cos(x)}$ increases from $0$ to $1$ for $x$ going from $0$ to $\pi/3$ and decreases from $1$ to $0$ for $x$ going from $\pi/3$ to $\pi$. Then the integral can be separated into
\begin{align}
I&=\int_0^\pi \arcsin{\left(\frac{\sin{x}}{\sqrt{5/4-\cos{x}}}\right)}\,dx\\
&=\int_0^{\pi/3} \arcsin{\left(\frac{\sin{x}}{\sqrt{5/4-\cos{x}}}\right)}\,dx+\int_{\pi/3}^\pi \arcsin{\left(\frac{\sin{x}}{\sqrt{5/4-\cos{x}}}\right)}\,dx
\end{align}
We define
\begin{equation}
u=\frac{\sin{x}}{\sqrt{5/4-\cos{x}}}
\end{equation}
or $u^2(5/4-\cos{x})=1-\cos^2x$.
Solving this equation for $\cos x$,
- for $0\le x\le\pi/3$, we have $\cos x=(u^2+\sqrt{u^4-5u^2+4})/2$
- for $\pi/3\le x\le\pi$, we have $\cos x=(u^2-\sqrt{u^4-5u^2+4})/2$.
Substituting $u$ into the integral, it comes
\begin{align}
I&=\int_0^1\arcsin(u)\frac{\sqrt{5-2 u^{2}-2 \sqrt{u^{4}-5 u^{2}+4}}}{\sqrt{u^{4}-5 u^{2}+4}}\,du+
\int_0^1\arcsin(u)\frac{\sqrt{5-2 u^{2}+2 \sqrt{u^{4}-5 u^{2}+4}}}{\sqrt{u^{4}-5 u^{2}+4}}\,du\\
&=\int_0^1\frac{\arcsin(u)}{\sqrt{u^{4}-5 u^{2}+4}}
f(u)\,du
\end{align}
where $f(u)=\sqrt{5-2 u^{2}-2 \sqrt{u^{4}-5 u^{2}+4}}+\sqrt{5-2 u^{2}+2 \sqrt{u^{4}-5 u^{2}+4}}$.
Now, it is possible to simplify:
\begin{align}
\left[f(u)\right]^2&=10-4u^2+2\sqrt{5-2 u^{2}-2 \sqrt{u^{4}-5 u^{2}+4}}\sqrt{5-2 u^{2}+2 \sqrt{u^{4}-5 u^{2}+4}}\\
&=4(4-u^2)
\end{align}
and
\begin{equation}
\sqrt{\frac{4(4-u^2)}{u^{4}-5 u^{2}+4}}=\frac{2}{\sqrt{1-u^2}}
\end{equation}
Then
\begin{align}
I&=2\int_0^1\frac{\arcsin(u)}{\sqrt{1-u^2}}\,du\\
&=\left.\arcsin^2(x)\right|_0^1\\
&=\frac{\pi^2}{4}
\end{align}
as expected.
