Is this a homework problem or is it some sort of practice question? If not, any possible symmetries coming from the formulation of the problem? If not, I don't know how far one can go, but let's see. It may help if you tell us how you derived the equation, weather it is an Euler-Lagrange equation of some sort. Maybe it's coming from a problem in differential geometry related to curves on surfaces?
Invert the function $x=x(t)$ and write it as a function $t=t(x)$. Then
$$\frac{d}{dt} = \frac{dx}{dt} \, \frac{d}{dx}$$
Form the metric $$\alpha^2 \left(\frac{dx}{dt}\right)^2 + x^4 \, \left(\frac{dy}{dt}\right)^2 = P^2$$ one gets
$$P^2 = \alpha^2 \left(\frac{dx}{dt}\right)^2 + x^4 \, \left(\frac{dx}{dt}\right)^2\left(\frac{dy}{dx}\right)^2 = \left(\frac{dx}{dt}\right)^2 \, \left(\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2\right)$$ and so
$$ \left(\frac{dx}{dt}\right)^2 = \frac{P^2}{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}$$
$$\frac{dx}{dt}= \frac{P}{\sqrt{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}}$$ Thus
$$\frac{d}{dt} = \left(\frac{P}{\sqrt{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}} \right)\,\, \frac{d}{dx}$$
Now the equation $$x y \, \frac{d^2 y}{dt^2} + 2 \, y \, \frac{dx}{dt}\frac{dy}{dt} + \alpha \frac{d^2x}{dt^2} + x \left(\frac{dy}{dt}\right)^2 = 0$$ can be written as
$$\frac{d}{dt}\left(x y \, \frac{dy}{dt} + \alpha \frac{dx}{dt}\right) + y \, \frac{dx}{dt}\frac{dy}{dt} = 0$$ Writing the latter equation in terms of $x$ as an independent variable
$$\left(\frac{P^2}{\sqrt{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}} \right)\,\, \frac{d}{dx} \left(\frac{x y \frac{dy}{dx} + \alpha}{\sqrt{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}}\right) + \frac{P^2 \, y \, \frac{dy}{dx}}{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2} = 0$$
$$ \frac{d}{dx} \left(\frac{x y \frac{dy}{dx} + \alpha}{\sqrt{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}}\right) + \frac{y \, \frac{dy}{dx}}{\sqrt{\alpha^2 + x^4 \, \left(\frac{dy}{dx}\right)^2}} = 0$$ In addition to that, one can also write $z(x) = \frac{1}{2} y(x)^2$ and so the equation becomes
$$ \frac{d}{dx} \left(\frac{x \frac{dz}{dx} + \alpha}{\sqrt{\alpha^2 + \frac{x^4}{2 \, z} \, \left(\frac{dz}{dx}\right)^2}}\right) + \frac{ \frac{dz}{dx}}{\sqrt{\alpha^2 + \frac{x^4}{2 \, z} \, \left(\frac{dz}{dx}\right)^2}} = 0$$
Alternatively, you can multiply the initial equation by $x$ and get
$$x^2 y \ddot{y} + 2xy \, \dot{x} \dot{y} + x^2 \dot{y}^2 + \alpha \, x \ddot{x} = 0$$ which becomes
$$0 = \frac{d}{dt} \big(x^2 y \dot{y}\big) + \alpha \, x \ddot{x} = \frac{d}{dt} \big(x^2 y \dot{y}\big) + \alpha \,\big( x \ddot{x} + \dot{x}^2\big) - \alpha \, \dot{x}^2 = \frac{d}{dt} \big(x^2 y \dot{y} + \alpha \, x \dot{x} \big) - \alpha \, \dot{x}^2$$
Now had your second equation been $$\alpha^2 \, \dot{x}^2 + x^4 y^2 \dot{y}^2 = P^2$$ then a miracle could have happened and since $x^2y\dot{y} = \sqrt{P^2 - \alpha^2 \, \dot{x}^2}$ the equation would have become
$$\frac{d}{dt} \Big(\sqrt{P^2 - \alpha^2 \, \dot{x}^2} + \alpha \, x \dot{x}\Big) - \alpha \, \dot{x}^2 = 0$$ which would have been a whole new story. This equation would have had much more potential.
Currently, you can end up with $$\frac{d}{dt} \Big( y \sqrt{P^2 - \alpha^2 \, \dot{x}^2} + \alpha \, x \dot{x}\Big) - \alpha \, \dot{x}^2 = 0$$ Maybe now one can write $y=y(t)$ as a function $t=t(y)$ and $x=x(y)$. Then
$$\frac{d}{dt} = \dot{y} \frac{d}{dy}$$ so
$$\dot{y} \frac{d}{dy}\left( y \sqrt{P^2 - \alpha^2 \, \dot{x}^2} + \alpha \, x \dot{x}\right) - \alpha \, \dot{x}^2 = 0$$
$$\dot{y} \frac{d}{dy}\left( y \sqrt{P^2 - \alpha^2 \, \dot{y}^2 \left(\frac{dx}{dy}\right)^2} + \alpha \, x \dot{y} \, \frac{dx}{dy}\right) - \alpha \, \dot{y}^2 \left(\frac{dx}{dy}\right)^2 = 0$$ Express $$\dot{y} = \frac{P}{\sqrt{\alpha^2\left(\frac{dx}{dy}\right)^2 + x^4}} = f\left(x,\frac{dx}{dy}\right)$$ so we get
$$f \frac{d}{dy}\left( y \sqrt{P^2 - \alpha^2 \, f^2 \left(\frac{dx}{dy}\right)^2} + \alpha \, x f \, \frac{dx}{dy}\right) - \alpha \, f^2 \left(\frac{dx}{dy}\right)^2 = 0$$
$$\frac{d}{dy}\left( y \sqrt{P^2 - \alpha^2 \, f^2 \left(\frac{dx}{dy}\right)^2} + \alpha \, x f \, \frac{dx}{dy}\right) - \alpha \, f \, \left(\frac{dx}{dy}\right)^2 = 0$$
$$\frac{d}{dy}\left( y \sqrt{P^2 - \alpha^2 \, f\left(x,\frac{dx}{dy}\right)^2 \left(\frac{dx}{dy}\right)^2} + \alpha \, x f\left(x,\frac{dx}{dy}\right) \, \frac{dx}{dy}\right) - \alpha \, f\left(x,\frac{dx}{dy}\right) \, \left(\frac{dx}{dy}\right)^2 = 0$$ This is again a $y$ inhomogeneous second order ODE of type $F\big(y, x, x'(y), x''(y)\big) = 0$.
Are you sure that the restriction is not $$\alpha^2 \, \dot{x}^2 + x^4 y^2 \dot{y}^2 = P^2 \, ? :)$$