Let me note that this is a Riccati equation. Therefore, given a solution for some initial conditions we can solve the equation (with the same parameters) for arbitrary initial conditions. For example, consider
$$y^2-y'-\beta^2=-\beta\cos x-\beta^2\cos^2 x \tag{1}$$
which is solved by $y(x)=\beta\sin x$. Substituting $\displaystyle y(x)=\beta\sin x+\frac{1}{v(x)}$ into (1) transforms the equation into a linear 1st order ODE
$$v'+2\beta\sin x \,v+1=0,$$
with the general solution
$$v(x)=\int_x^ae^{2\beta(\cos x-\cos t)}dt,$$
where $a$ is an arbitrary integration constant.
Riccati property also implies that the change of variables $\displaystyle y=-\frac{u'}{u}$ in the equation $y^2-y'+f=0$ transforms it into a linear 2nd order ODE
$$u''+fu=0, \tag{2}$$
where in our case $f(x)=A\cos x+B\cos^2 x+C$. Obviously, for $A=0$ or $B=0$ (studied cases) this can be transformed into the Mathieu's equation. This provides some insight: let us make the change of variables $t=\cos\frac{x}{2}$. It will transform (2) into
$$(1-t^2)u_{tt}-tu_t+4\underbrace{\left[A(2t^2-1)+B(2t^2-1)^2+C\right]}_{Q(t^2)}u=0\tag{3}$$
This ODE has 3 singularities on $\mathbb{P}^1$: two regular singular points at $\pm1$ and one irregular at $\infty$. The solutions listed in the question have the following form in terms of $u(t)$:
\begin{align}
u_1(t)=&e^{2\beta t^2},\\
u_2(t)=&te^{2\beta t^2},\\
u_3(t)=&\sqrt{1-t^2}e^{2\beta t^2},\\
u_4(t)=&t\sqrt{1-t^2}e^{2\beta t^2}.
\end{align}
Here $u_{2,3}$ correspond to $\pm$ sign in the second family.
Apparently one can construct an infinite number of solutions of each of the four types. Let me first give one more example:
$$u_5(t)=\left(t^2+\sigma\right)e^{2\beta t^2},$$
which solves (3) for the following parameter values:
\begin{align}
A=3\beta,\qquad B=\beta^2,\qquad C=1-2\beta-\beta^2-4\beta\sigma,\qquad
\beta=\frac{1+2\sigma}{8\sigma(1+\sigma)}.
\end{align}
In terms of the initial function $y(x)$, this one-parameter family of solutions is written as
$$y_5(x)=\frac{\sin x}{8}\left[\frac{1}{\sigma}+\frac{1}{1+\sigma}+
\frac{8}{1+2\sigma+\cos x}\right].\tag{4}$$
Next let us generalize this result and show that for any positive integer $n$ there is a one-parameter family of solutions of the form
$$u(t)=P_{n}(t^2)\,e^{2\beta t^2},\tag{5}$$
where $P_{n}(z)$ is a $n$-th degree polynomial in $z$ with coefficients depending on $\beta$. Indeed, write
$$P_n(z)=z^n+\sum_{k=0}^{n-1}a_kz^k,\tag{6}$$
then (3) gives the following equation for $P_n(z)$:
\begin{align}z(1-z)P_n''(z)+\left(\frac12+(4\beta-1)z-4\beta z^2\right)P_n'(z)+\\
+\biggl(Q(z)-\beta\left[4\beta z^2+2(1-2\beta)z-1\right]\biggr)P_n(z)=0. \tag{7}
\end{align}
Recall that $Q(z)$ is a 2nd degree polynomial, hence the left side of (7) is a $(n+2)$th degree polynomial. Thus we have to satisfy $n+3$ identities with $n+4$ parameters $A,B,C,\beta,a_0,\ldots,a_{n-1}$, and the statement follows.
In particular, substituting the expansion (6) into (7) and setting the coefficient of $z^{n+2}$ to $0$, we find the constraint $B=\beta^2$. Similarly equating the coefficient of $z^{n+1}$ to $0$, we get $A=(2n+1)\beta$. Hence the one-parameter family (5) is characterized by
$$A=(2n+1)\beta,\qquad B=\beta^2.$$
Similar reasoning allows to establish the following
Theorem. For any $n\in\mathbb{Z}_{\ge 0}$ there are four one-parameter families of solutions of (3) of the form
\begin{align}
u^{\mathrm{(I)}}(t)\;\;=&P_n^{\mathrm{(I)}}(t^2)\,e^{2\beta t^2},\\
u^{\mathrm{(II)}}(t)\;=&tP_n^{\mathrm{(II)}}(t^2)\,e^{2\beta t^2},\\
u^{\mathrm{(III)}}(t)=&\sqrt{1-t^2}P_n^{\mathrm{(III)}}(t^2)\,e^{2\beta t^2},\\
u^{\mathrm{(IV)}}(t)\,=&t\sqrt{1-t^2}P_n^{\mathrm{(IV)}}(t^2)\,e^{2\beta t^2}.
\end{align}
Here $P_n^{\mathrm{(k)}}(z)$ with $k=\mathrm{I,II,III,IV}$ are $n$th degree polynomials of the form (6) whose coefficients can be determined from (3). The appropriate parameter values are
\begin{align}
(A,B)^{\mathrm{(I)}}\;\;=&\Bigl((2n+1)\beta,\beta^2\Bigr),\\
(A,B)^{\mathrm{(II)}}\;=&\Bigl((2n+2)\beta,\beta^2\Bigr),\\
(A,B)^{\mathrm{(III)}}=&\Bigl((2n+2)\beta,\beta^2\Bigr),\\
(A,B)^{\mathrm{(IV)}}\,=&\Bigl((2n+3)\beta,\beta^2\Bigr).
\end{align}
The solutions mentioned in the question correspond to setting $n=0$ in the above.
