Here is my attempt at a solution. I did not get your very nice answer of $\frac{5\pi^4}{2304}$, but if yours and my answers are equivalent, it will mean we have found a value for $\operatorname{Re} \operatorname{Li}_4 (1 + i)$ (is a closed-form value known for this quantity?).
Set
$$I = \int_0^{\frac{\pi}{4}} \ln^2 \tan x (4 \cot x \ln \sec x - x) \, dx.$$
Enforcing a substitution of $x \mapsto \arctan x$ leads to
$$I = 2 \int_0^1 \frac{\ln^2 x}{x} \frac{\ln (1 + x^2)}{1 + x^2} \, dx - \int_0^1 \frac{\ln^2 x \arctan x}{1 + x^2} \, dx = 2I_1 - I_2.$$
First integral $I_1$
Making use of the following generating function for the harmonic numbers, namely
$$\frac{\ln (1 + x^2)}{1 + x^2} = -\sum_{n = 1}^\infty (-1)^n H_n x^{2x}.$$
we have
\begin{align}
I_1 &= -\sum_{n = 1}^\infty (-1)^n H_n \int_0^1 x^{2n - 1} \ln^2 x \, dx\\
&= -\sum_{n = 1}^\infty (-1)^n H_n \frac{d^2}{ds^2} \left [\int_0^1 x^{2n + s -1} \, dx \right ]_{s = 0}\\
&= -\sum_{n = 1}^\infty (-1)^n H_n \frac{d^2}{ds^2} \left [\frac{1}{2n + s} \right ]_{s = 0}\\
&= -\frac{1}{4} \sum_{n = 1}^\infty \frac{(-1)^n H_n}{n^3}.
\end{align}
Second integral $I_2$
Taking the Cauchy product between the Maclaurin series expansions for $\arctan x$ and $\frac{1}{1 + x^2}$ one finds
$$\frac{\arctan x}{1 + x^2} = \sum_{n = 0}^\infty (-1)^n \left (H_{2n + 1} - \frac{1}{2} H_n \right ) x^{2n + 1}.$$
Thus
\begin{align}
I_2 &= \sum_{n = 0}^\infty (-1)^n \left (H_{2n + 1} - \frac{1}{2} H_n \right ) \int_0^1 x^{2n + 1} \ln^2 x \, dx\\
&= \sum_{n = 0}^\infty (-1)^n \left (H_{2n + 1} - \frac{1}{2} H_n \right ) \frac{d^2}{ds^2} \left [\int_0^1 x^{2n + s + 1} \, dx \right ]_{s = 0}\\
&= \sum_{n = 0}^\infty (-1)^n \left (H_{2n + 1} - \frac{1}{2} H_n \right ) \frac{d^2}{ds^2} \left [\frac{1}{2n + s + 2} \right ]_{s = 0}\\
&= \frac{1}{4} \underbrace{\sum_{n = 0}^\infty \frac{(-1)^n}{(n + 1)^3} \left (H_{2n + 1} - \frac{1}{2} H_n \right )}_{n \, \mapsto \, n - 1}\\
&= \frac{1}{4} \sum_{n = 1}^\infty \frac{(-1)^{n - 1}}{n^3} \left (H_{2n - 1} - \frac{1}{2} H_{n - 1} \right )\\
&= -\frac{1}{4} \sum_{n = 1}^\infty \frac{(-1)^n}{n^3} \left (H_{2n} - \frac{1}{2n} \right ) + \frac{1}{8} \sum_{n = 1}^\infty \frac{(-1)^n}{n^3} \left (H_n - \frac{1}{n} \right )\\
&= -\frac{1}{4} \sum_{n = 1}^\infty \frac{(-1)^n H_{2n}}{n^3} + \frac{1}{8} \sum_{n = 1}^\infty \frac{(-1)^n H_n}{n^3}
\end{align}
Main integral $I$
So for the main integral $I$ we have
$$I = -\frac{5}{8} \sum_{n = 1}^\infty \frac{(-1)^n H_n}{n^3} + \frac{1}{4} \sum_{n = 1}^\infty \frac{(-1)^n H_{2n}}{n^3}.$$
Dealing with these two Euler sums, their values can be found from the following generating function
\begin{align}
\sum^\infty_{n=1}\frac{H_n}{n^3}x^n
&=2{\rm Li}_4(x)+{\rm Li}_4\left(\tfrac{x}{x-1}\right)-{\rm Li}_4(1-x)-{\rm Li}_3(x)\ln(1-z)-\frac{1}{2}{\rm Li}_2^2\left(\tfrac{x}{x-1}\right)\\
&+\frac{1}{2}{\rm Li}_2(x)\ln^2(1-x)+\frac{1}{2}{\rm Li}_2^2(x)+\frac{1}{6}\ln^4(1-x)-\frac{1}{6}\ln{x}\ln^3(1-x)\\
&+\frac{\pi^2}{12}\ln^2(1-x)+\zeta(3)\ln(1-x)+\frac{\pi^4}{90},\tag1
\end{align}
which is proved in this answer here.
Setting $x = -1$ in (1) gives
\begin{align}
\sum^\infty_{n=1}\frac{(-1)^nH_n}{n^3}=2{\rm Li}_4\left(\tfrac{1}{2}\right)-\frac{11\pi^4}{360}+\frac{7}{4}\zeta(3)\ln{2}-\frac{\pi^2}{12}\ln^2{2}+\frac{1}{12}\ln^4{2},
\end{align}
while setting $x = i$ in (1) gives
\begin{align}
\frac{1}{4} \sum_{n = 1}^\infty \frac{(-1)^n H_{2n}}{n^3} &= 2 \sum_{n = 1}^\infty \frac{(-1)^n H_{2n}}{(2n)^3}\\
&= 2 \operatorname{Re} \sum_{n = 1}^\infty \frac{H_n}{n^3} i^n\\
&= -4 \operatorname{Re} \operatorname{Li}_4(1 + i) + \frac{29 \pi^4}{1152} + \frac{35}{32} \zeta (3) \ln 2 + \frac{\pi^2}{32} \ln^2 2.
\end{align}
Substituting these two values for the Euler sums back into the expression for the integral $I$ gives a final answer of
$$I = -\frac{5}{4} \operatorname{Li}_4 \left (\frac{1}{2} \right ) - 4 \operatorname{Re} \operatorname{Li}_4 (1 + i) + \frac{17}{384} \pi^4 + \frac{\pi^2}{12} \ln^2 2 - \frac{5}{96} \ln^4 2.$$
So, this leads one to the following conjecture. Does?
$$\operatorname{Re} \operatorname{Li}_4 (1 + i) = -\frac{5}{16} \operatorname{Li}_4 \left (\frac{1}{2} \right ) + \frac{97}{9216} \pi^4 + \frac{\pi^2}{48} \ln^2 2 - \frac{5}{384} \ln^4 2$$
Update
The conjecture is true! A proof of this can be found here. So one does indeed have
$$\int_0^{\frac{\pi}{4}} \ln^2 \tan x (4 \cot x \ln \sec x - x) \, dx = \frac{5 \pi^4}{2304}.$$
It would of course be nice to find a simple (simpler?) solution to this integral that, unlike my solution, does not rely on knowing the value for $\operatorname{Re} \operatorname{Li}_4 (1 + i)$.