This integral appears as an exercise on page 857 of the textbook Classical Complex Analysis: A Geometric Approach, Volume 1 by I-Hsiung Lin.
You're told to integrate the function $$ f(z) = \log^{2}(z) \Big( \frac{1-z}{1+z} \Big)^{1/2}$$ around an closed indented contour that runs just above the real axis from $z=-1$ to $z=1$ and then along the upper half of the unit circle.
Since the contributions from the indentations around the branch points at $z=0, z=1$, and $z=-1$ can be shown to be vanishing small, we get (using the principal branch of the logarithm throughout),
$$ \int_{-1}^{0} \left(\log|x| + i \pi\right)^{2}\left( \frac{1-x}{1+x} \right)^{1/2} \, dx + \int_{0}^{1} \log^{2}(x) \left(\frac{1-x}{1+x} \right)^{1/2} \, dx $$
$$ \ + i \int_{0}^{\pi} \log^{2}(e^{it}) \left(\frac{1-e^{i t}}{1+e^{i t}} \right)^{1/2} e^{it} \, dt = 0$$
Looking at each piece separately,
$$ \begin{align} \int_{-1}^{0} \left(\log|x| + i \pi \right)^{2} \left( \frac{1-x}{1+x} \right)^{\frac{1}{2}} \, dx &= \int_{0}^{1} \left(\log x + i \pi \right)^{2} \left(\frac{1+x}{1-x} \right)^{1/2} \, dx \\ &= \int_{0}^{1}\left(\log^{2} (x) + 2 \pi i \log x - \pi^{2} \right) \frac{1+x}{\sqrt{1-x^{2}}} \, dx, \end{align}$$
$$\int_{0}^{1} \log^{2}(x) \left(\frac{1-x}{1+x} \right)^{1/2} \ dx = \int_{0}^{1} \log^{2}(x) \frac{1-x}{\sqrt{1-x^{2}}} \, dx, $$
and
$$ \begin{align} i \int_{0}^{\pi} \log^{2}(e^{it}) \Big(\frac{1-e^{i t}}{1+e^{i t}} \Big)^{1/2} e^{it} \ dt &= -i \int_{0}^{\pi} t^{2} \sqrt{-i \tan t/2} \ e^{it} \ dt \\ &= - i e^{-i \pi/4} \int_{0}^{\pi} t^{2} \sqrt{\tan t/2} \ e^{it} \ dt \end{align}$$
So we have
$$ e^{-i \pi /4} \int_{0}^{\pi} t^{2} \sqrt{\tan t/2} \, e^{it} \, dt = -2i \int_{0}^{1} \frac{\log^{2}(x)}{\sqrt{1-x^{2}}} \, dx + 2 \pi \int_{0}^{1} \frac{\log x}{\sqrt{1-x^{2}}} \, dx $$
$$ + 2 \pi \int_{0}^{1} \frac{x \log x}{\sqrt{1-x^{2}}} \, dx +i \pi^{2} \int_{0}^{1} \frac{1}{\sqrt{1-x^{2}}} \, dx +i \pi^{2} \int_{0}^{1} \frac{x}{\sqrt{1-x^{2}}} \, dx$$
We can use the fact that
$$ \int_{0}^{1} \frac{x^{a}}{\sqrt{1-x^{2}}} \ dx = \frac{1}{2} \int_{0}^{1} u^{a/2+1/2-1} (1-u)^{1/2-1} \ du = \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right)$$
along with properties of the beta function (entries 23-26) to evaluate the first three integrals on the right side of the equation.
$$ \begin{align} \int_{0}^{1} \frac{\log^{2}(x)}{\sqrt{1-x^{2}}} \, dx &= \frac{d^{2}}{da^{2}} \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right) \Bigg|_{a=0} \\ &= \frac{1}{8} B \left( \frac{1}{2}, \frac{1}{2} \right) \left[ \left( \psi \left(\frac{1}{2} \right) - \psi(1) \right)^{2} + \psi_{1}\left(\frac{1}{2}\right) - \psi_{1}(1) \right] \\ &= \frac{\pi^{3}}{24} + \frac{\pi \log^{2} 4}{8} \end{align}$$
$$ \begin{align} \int_{0}^{1} \frac{\log x}{\sqrt{1-x^{2}}} \, dx &= \frac{d}{da} \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right) \Bigg|_{a=0} \\ &= \frac{1}{4} B \left( \frac{1}{2}, \frac{1}{2} \right) \left( \psi\left(\frac{1}{2}\right) - \psi(1) \right) \\ &= - \frac{\pi \log 2}{2} \end{align}$$
$$ \begin{align} \int_{0}^{1} \frac{x \log x}{\sqrt{1-x^{2}}} \, dx &= \frac{d}{da} \frac{1}{2} B \left( \frac{1}{2}, \frac{a}{2}+ \frac{1}{2} \right) \Bigg|_{a=1} \\ &= \frac{1}{4} B \left( \frac{1}{2}, 1 \right) \left( \psi(1) - \psi\left(\frac{3}{2}\right) \right) \\ &= \log 2 -1 \end{align}$$
Therefore,
$$ \int_{0}^{\pi} t^{2} \sqrt{\tan t/2} \ e^{it} \ dt = e^{i \pi /4} \left( -\frac{i \pi^{3}}{12} - \frac{i\pi \log^{2} 4}{4} - \pi^{2} \log 2 + 2 \pi \log 2 - 2 \pi + \frac{i\pi^{3}}{2} +i \pi^{2} \right)$$
Now just make the substitution $u = \frac{t}{2}$, and then equate the imaginary parts on both sides of the equation.