Hint: $\sin x = \dfrac{\tan x}{\sec x}$ and $\sec^2x = 1 + \tan^2x$.
$$\begin{align}
\dfrac1{(\sin x+a\sec x)^2} &= \dfrac1{\left(\dfrac{\tan x}{\sec x}+a\sec x\right)^2} \\
&= \dfrac{\sec^2x}{(\tan x + a\sec^2x)^2} \\
&= \dfrac{\sec^2x}{\tan^2x + 2a\tan x\sec^2x + a^2\sec^4x} \\
&= \dfrac{\sec^2x}{\tan^2x + 2a\tan x(1 + \tan^2x)+ a^2(1 + \tan^2x)^2}
\end{align}$$
Therefore,
$$\int\dfrac1{(\sin x+a\sec x)^2}\,\mathrm dx = \int\dfrac{\sec^2x}{\tan^2x + 2a\tan x(1 + \tan^2x)+ a^2(1 + \tan^2x)^2}\,\mathrm dx$$
Substitute $u = \tan x$, $\implies\mathrm du = \sec^2x\,\mathrm dx$. Now,
$$\begin{align}\int\dfrac{\sec^2x}{\tan^2x + 2a\tan x(1 + \tan^2x)+ a^2(1 + \tan^2x)^2}\,\mathrm dx &= \int \dfrac1{u^2 + 2au(1 + u^2) + a^2(1 + u^2)^2}\,\mathrm du\end{align}$$
From here onward, you would need to factor the denominator and perform partial fraction decomposition. Then, you will be able to apply linearity to integrate individual fractions. Finally, undo substitution to get the final result.
Edit # 1:
$$\begin{align}
u^2 + 2au(1 + u^2) + a^2(1 + u^2)^2 &\equiv u^2 + 2\cdot (u)\cdot\left(a(1 + u^2)\right) + \left(a(1 + u^2)\right)^2 \\
&= \left(u + a(1 + u^2)\right)^2
\end{align}$$
Factorising $u + a(1 + u^2)$:
$$u + a(1 + u^2) \equiv au^2 + u + a$$
Completing square,
$$\begin{align}
au^2 + u + a &= au^2 + u + \dfrac1{4a} - \dfrac1{4a} + a \\
&= \left(\sqrt{a}u + \dfrac1{2\sqrt{a}}\right)^2 - \dfrac1{4a} + a \\
&= \dfrac{(2au + 1)^2}{4a} - \dfrac{1 - 4a^2}{4a} \\
&= \dfrac{(2au + 1)^2 - \left(\sqrt{1 - 4a^2}\right)^2}{4a} \\
&= \dfrac{\left(2au + 1 + \sqrt{1 - 4a^2}\right)\left(2au + 1 - \sqrt{1 - 4a^2}\right)}{4a}
\end{align}$$
Therefore,
$$\begin{align}
\int \dfrac1{u^2 + 2au(1 + u^2) + a^2(1 + u^2)^2}\,\mathrm du &= \operatorname{\large\int} \dfrac1{\left(au^2 + u + a\right)^2}\,\mathrm du\\
&= \operatorname{\Large\int} \dfrac{1}{\left(\left(2au + 1 + \sqrt{1 - 4a^2}\right)\left(2au + 1 - \sqrt{1 - 4a^2}\right)/4a\right)^2}\,\mathrm du \\
&= \operatorname{\Large\int} \dfrac{16a^2}{\left(2au + 1 + \sqrt{1 - 4a^2}\right)^2\left(2au + 1 - \sqrt{1 - 4a^2}\right)^2}\,\mathrm du
\end{align}$$