When you divide the numerator and the denominator by $\cos^2 x$, you should get $$\displaystyle \int^{\pi/4}_0 \dfrac{\sec^2 x dx}{\sec^2 x - 3\sin^2 x}$$ instead of $$\displaystyle \int^{\pi/4}_0 \dfrac{\sec^2 x dx}{\sec^2 x - 3 \tan^2 x}$$
because
$$\begin{array}{rcll}
\displaystyle \int^{\pi/4}_0 \dfrac{dx}{1 - 3\sin^2 x \cos^2 x} &=& \displaystyle\int^{\pi/4}_0 \dfrac{\frac{dx}{\cos^2 x}}{\frac{1 - 3\sin^2 x\cos^2 x}{\cos^2 x}}\\
&=&\displaystyle\int^{\pi/4}_0 \dfrac{\sec^2 x dx}{\frac{1}{\cos^2 x} - \frac{3\sin^2 x\cos^2 x}{\cos^2 x}}\\
&=& \displaystyle\int^{\pi/4}_0 \dfrac{\sec^2 x dx}{\sec^2 x - 3 \sin^2 x}
\end{array}$$
Denote the original integral as $I$.
For me, at this point:$$\displaystyle \int^{\pi/4}_0 \dfrac{dx}{1 - 3\sin^2 x \cos^2 x}$$
I would use double angle formula to write $3 \sin^2 x \cos^2 x$ in terms of $2x$.
From $\sin 2x = 2 \sin x \cos x$, square both sides:$$\sin^2 2x = 4\sin^2 x \cos^2 x$$Multiply by $\dfrac{3}{4}$:$$\dfrac{3}{4}\sin^2 2x = 3\sin^2 x\cos^2 x$$
The integral becomes $$\displaystyle \int^{\pi/4}_0 \dfrac{dx}{1 - \frac{3}{4}\sin^2 2x}$$
Next, let $u = 2x$,$\;du = 2dx$.$$\begin{array}{rcll}
I&=&\displaystyle \int^{\pi/4}_0 \dfrac{dx}{1 - \frac{3}{4}\sin^2 2x}\\ &=& \displaystyle \int^{\pi/2}_0 \dfrac{\frac{du}{2}}{1 - \frac{3}{4}\sin^2 u}\\
&=&2 \displaystyle \int^{\pi/2}_0 \dfrac{du}{4 - 3\sin^2 u}
\end{array}$$
Using the fact that $\sin^2 u = \dfrac{1 - \cos 2u}{2}$,
$$\begin{array}{rcll}
I&=&2\displaystyle\int^{\pi/2}_0 \dfrac{du}{4 - 3\sin^2 u}
\\&=& 2\displaystyle \int^{\pi/2}_0 \dfrac{du}{4 - 3 \left(\frac{1 - \cos 2u}{2}\right)}
\\&=& 4\displaystyle \int^{\pi/2}_0 \dfrac{du}{8 - 3(1 - \cos 2u)}
\\&=& 4\displaystyle \int^{\pi/2}_0 \dfrac{du}{5 + 3\cos 2u}
\end{array}$$
Next, noticing that $\cos 2u = \dfrac{1 - \tan^2 u}{1 + \tan^2 u}$,
$$\begin{array}{rcll}
I&=& 4\displaystyle \int^{\pi/2}_0 \dfrac{du}{5 + 3\cos 2u}
\\&=& 4 \displaystyle \int^{\pi/2}_0 \dfrac{du}{5 + 3\left(\frac{1 - \tan^2 u}{1 + \tan^2 u}\right)}
\\&=& 4 \displaystyle \int^{\pi/2}_0 \dfrac{\sec^2 u du}{5(1 + \tan^2 u) + 3\left(1 - \tan^2 u\right)}
\\&=& 4 \displaystyle \int^{\pi/2}_0 \dfrac{\sec^2 u du}{8 + 2 \tan^2 u}
\\&=& 2 \displaystyle \int^{\pi/2}_0 \dfrac{d\left(\tan u\right)}{4 + \tan^2 u}
\end{array}$$
Using the substitution $t = \tan u$,
$$\begin{array}{rcll}
I&=& 2 \displaystyle \int^{\pi/2}_0 \dfrac{d\left(\tan u\right)}{4 + \tan^2 u}
\\&=& 2 \displaystyle \lim_{\varepsilon \to 0^{+}}\int^{1/\varepsilon}_0 \dfrac{dt}{4 + t^2}
\\&=& 2 \displaystyle \lim_{\varepsilon \to 0^{+}}\left[\dfrac{1}{2} \arctan\left(\dfrac{t}{2}\right)\right]^{1/\varepsilon}_0
\\&=& 2 \displaystyle \lim_{\varepsilon \to 0^{+}} \left(\dfrac{1}{2}\arctan\left(\dfrac{1}{2\varepsilon}\right)\right)
\\&=& \displaystyle \lim_{\varepsilon \to 0^{+}} \arctan\left(\dfrac{1}{2\varepsilon}\right)
\\&=& \color{red}{\boxed{ \dfrac{\pi}{2} }}
\end{array}$$
