How to find the equation of the conic before applying the rotation? Given the rotation matrix:
$$Q=\begin{bmatrix}\frac{2}{\sqrt{5}}& \frac{1}{\sqrt{5}}\\-\frac{1}{\sqrt{5}}& \frac{2}{\sqrt{5}}\end{bmatrix},$$ I want to find the equation of the conic $C$ given by $$x^2+4xy+4y^2-10\sqrt{5}x=0,$$ which I know is a parabola, before applying the rotation to it. 
I was thinking about solving the system of equation $$\begin{cases}x=\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\\ y=-\frac{1}{\sqrt{5}}\tilde{x}+\frac{2}{\sqrt{5}}\tilde{y}\end{cases},$$ and then substituting $x$ and $y$ in the initial equation of the conic. The result should be $$5\tilde{y}^2-20\tilde{x}-10\tilde{y}=0,$$ but it seems that my results are different.  
 A: Your approach is entirely correct; plugging in your substitutions into the first equation for $C$ yields the result you claim it should be:
\begin{eqnarray*}
x^2+4xy+4y^2-10\sqrt{5}x&=&
\left(\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\right)^2+4\left(\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\right)\left(-\frac{1}{\sqrt{5}}\tilde{x}+\frac{2}{\sqrt{5}}\tilde{y}\right)\\
&\ &+4\left(-\frac{1}{\sqrt{5}}\tilde{x}+\frac{2}{\sqrt{5}}\tilde{y}\right)^2-10\sqrt{5}\left(\frac{2}{\sqrt{5}}\tilde{x}+\frac{1}{\sqrt{5}}\tilde{y}\right)\\
&=&\frac{1}{5}(4\tilde{x}^2+4\tilde{x}\tilde{y}+\tilde{y}^2)+\frac{4}{5}(-2\tilde{x}^2+3\tilde{x}\tilde{y}+2\tilde{y}^2)\\
&\ &+\frac{4}{5}(\tilde{x}^2-4\tilde{x}\tilde{y}+4\tilde{y}^2)-10(2\tilde{x}+\tilde{y})\\
&=&5\tilde{y}^2-20\tilde{x}-10\tilde{y}.
\end{eqnarray*}
You seem to have made a mistake in your calculation.
A: Basically, it looks like you’re getting the transformation backwards. If you have a coordinate transformation $\mathbf x'=Q\mathbf x$, to get the transformed equation you have to substitute for $\mathbf x$ in the original equation, i.e., apply the inverse transformation to $x$ and $y$ in the equation. So, if the original coordinate transformation was given by $Q$, you now want to transform by $Q^{-1}$, which means that you substitute $Q(x,y)^T$ for $(x,y)$ in the equation that you have.
A: Suppose we have a conic with the equation $P(x,y) = 0.$
Next, you apply the rotation $Q$ to the conic,
so if $(\tilde x,\tilde y)$ is a point on the conic before rotation,
it is taken to the point 
$$
(x,y)=\left(\tfrac{2}{\sqrt5}\tilde{x}+\tfrac{1}{\sqrt5}\tilde{y},-\tfrac{1}{\sqrt5}\tilde{x}+\tfrac{2}{\sqrt5}\tilde{y}\right) \tag1
$$
after rotation.
Moreover, after applying the rotation, the rotated conic has the equation
$$ x^2+4xy+4y^2-10\sqrt{5}x=0. \tag2$$
If you now use Equation $(1)$ to provide the values of $x$ and $y$ 
in terms of $\tilde x$ and $\tilde y,$
you can make those substitutions for $x$ and $y$ in Equation $(2),$
and then you have an equation that must be satisfied by any point
$(\tilde x,\tilde y)$ on the original conic.
Try it.
The fact that one of your steps was to "solve" Equation $(1)$ suggests that your substitution was actually for a rotation by the correct angular magnitude, but in the opposite direction from what you needed.
