reduction of $6xy + 8 y^2 -12x-26y + 11 = 0$ to canonical one of a second-order curve I have this polynomial 
$$
6xy + 8 y^2 -12x-26y + 11 = 0
$$
and I need to reduce it to a canonical equation of a second-order curve. The correct answer from the textbook is that it is a hyperbola 
$$
\frac{X^2}{1} - \frac{Y^2}{9} = 1
$$
and the coordinate system must be rotated to an angle equal to 
$$
\arctan{3}
$$
and the origin of the new coordinate system must be moved to the point $O'(-1,2)$. I checked this in Mathematica® program, so I think it is safe to assume that there are no errors in the textbook.

And now about the problem itself. I calculated the angle of rotation according to the formula in my textbook. It is (the expression above is the example of a polynom to explain indexes like $a,b,c$ etc.)
$$
ax^2 + 2bxy + cy^2 + 2dx + 2ey +g = 0 \\
\cot{2\alpha} = \frac{a-c}{2b}
$$
I solved this equation and got the correct answer: $\alpha = \arctan{3}$
Then I substituted $x$ and $y$ variables with their values according to the formulae:
$$
x = \cos{\alpha}x' - \sin{\alpha}y'\\
y = \sin{\alpha}x' + \cos{\alpha}y'
$$
Let me show this incrementally for convenience
$$
6xy + 8 y^2 =\\
6(\cos{\alpha}x' - \sin{\alpha}y')(\sin{\alpha}x' + \cos{\alpha}y') + 8(\sin{\alpha}x' + \cos{\alpha}y')^2 =\\
6(\sin{\alpha}\cos{\alpha} * x'^2 +\cos^2{\alpha} * x'y' -\sin^2{\alpha} * x'y' - \sin{\alpha}\cos{\alpha} * y'^2) + 8(\sin^2{\alpha} * x'^2 + 2\sin{\alpha}\cos{\alpha} * x'y' + \cos^2{\alpha} * y'^2) =\\
6\sin{\alpha}\cos{\alpha} * x'^2 + 8\sin^2{\alpha} * x'^2 - 6\sin{\alpha}\cos{\alpha} * y'^2 + 8\cos{\alpha}^2 * y'^2 + 6\cos{\alpha}^2 * x'y' -6\sin^2{\alpha} * x'y' + 16\sin{\alpha}\cos{\alpha} * x'y' = \\
(6\sin{\alpha}\cos{\alpha} + 8\sin^2{\alpha})x'^2 + (- 6\sin{\alpha}\cos{\alpha} + 8\cos^2{\alpha})y'^2 + (6\cos{\alpha}^2 -6\sin^2{\alpha} + 16\sin{\alpha}\cos{\alpha})x'y'
$$
the factor at $x'y'$ evaluates to 0 given $\alpha = \arctan{3}$, I checked this in Mathematica, so from the part above only this remains:
$$
(6\sin{\alpha}\cos{\alpha} + 8\sin^2{\alpha})x'^2 + (- 6\sin{\alpha}\cos{\alpha} + 8\cos^2{\alpha})y'^2
$$
And the remaining part is:
$$
-12x-26y + 11 = \\
-12(\cos{\alpha}*x' - \sin{\alpha}*y') - 26(\sin{\alpha}*x' + \cos{\alpha}*y') + 11 = \\
-12\cos{\alpha}*x' + 12\sin{\alpha}*y' -26\sin{\alpha}*x' - 26\cos{\alpha}*y' + 11 = \\
(-12\cos{\alpha} - 26\sin{\alpha})*x' + (12\sin{\alpha} - 26\cos{\alpha})*y' + 11 = \\
2((-6\cos{\alpha} - 13\sin{\alpha}))*x' + 2(6\sin{\alpha} - 13\cos{\alpha})*y' + 11
$$
So now this curve has the next equation in the rotated coordinate system (let me use $x$ and $y$ instead of $x'$ and $y'$):
$$
\alpha = \arctan{3}\\
a = 6\sin{\alpha}\cos{\alpha} + 8\sin^2{\alpha} \\
c = - 6\sin{\alpha}\cos{\alpha} + 8\cos^2{\alpha} \\
d = -6\cos{\alpha} - 13\sin{\alpha} \\
e = 6\sin{\alpha} - 13\cos{\alpha}\\
g = 11\\
ax^2 + cy^2 + 2dx  + 2ey + g = 0
$$
Then it is stated in my textbook that one can further reduce this equation by moving the origin of a coordinate system to the point $O(-\frac{d}{a}, -\frac{e}{c})$
And this is the place where I can't get the correct answer (which is $O(-1,2)$), cause this $-\frac{d}{a}$ gives me 1.58114 instead of -1 and this $-\frac{e}{c}$ gives 1.58114 instead of 2. 
I've checked this in Mathematica.


Could anyone explain me what where I'm wrong, please?
Thank you in advance.
 A: I always try to complete the squares:
$$6xy+8y^2 −12x−26y+11=0$$
$$\left[ 8y^2+2.2 \sqrt{2}y\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)\right]-12x+11=0 $$
$$\left[ 8y^2+2.2 \sqrt{2}y\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)+\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)^2\right]-\left(\frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}}\right)^2-12x+11=0 $$
$$\left[2\sqrt{2}y + \frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}} \right]^2-\frac{1}{8}(9x^2+18x+81)=0 $$
$$\left[2\sqrt{2}y + \frac{3x}{2\sqrt{2}}-\frac{13}{2\sqrt{2}} \right]^2-\frac{1}{8}(3x+3)^2=9 $$
$$\frac{1}{8}(8y+3x-13)^2-\frac{1}{8}(3x+3)^2=9 $$
$$(8y+3x-13)^2-(3x+3)^2=72$$
$$(8y+3x-13)^2-\frac{(x+1)^2}{9}=72$$
$$X^2-\frac{Y^2}{9}=72$$
A: Hint:
When $\tan \alpha = 3,\,$ then $\cos \alpha = \frac{1}{\sqrt{10}},\,\sin \alpha=\frac{3}{\sqrt{10}}\Rightarrow$
$6xy + 8 y^2 -12x-26y + 11 = 0$
$x = x'\frac{1}{\sqrt{10}} - y'\frac{3}{\sqrt{10}}\\
y = x'\frac{3}{\sqrt{10}} + y'\frac{1}{\sqrt{10}}$
$\Rightarrow 9x'^2-y'^2-9\sqrt{10}x'+\sqrt{10}y'+11=0\Rightarrow \frac{(x'-\sqrt{5/2})^2}{1}-\frac{(y'-\sqrt{5/2})^2}{9}=1$
$\Rightarrow \frac{X^2}{1}-\frac{Y^2}{9}=1$
A: Finally I have found where I was wrong.
I was getting wrong results for this expression $O(-\frac{d}{a}, -\frac{e}{c})$ - $(1.58114, 1.58114)$ instead of $(-1,2)$  because it was giving me coordinates of hyperbola's center in the already rotated coordinate system instead of the original one. To get the correct coordinates of the new system in the old system I need to perform a reverse operation, i.e. express $x,y$ of the new origin in the original system, that is perform rotation to an angle $-\alpha$ and shifting to the point $O(\frac{d}{a}, \frac{e}{c})$. Generally I need to take the new system as an "old" one.
When I did this, given 
$$
\tan{\alpha} = 3 \\
\sin{-\alpha} = - \sin{\alpha} = - \frac{\tan{\alpha}}{\sqrt{1 + \tan^2{\alpha}}} = - \frac{3}{\sqrt{10}}, \\
\cos{-\alpha} = \cos{\alpha} = \frac{1}{\sqrt{1 + \tan^2{\alpha}}} = \frac{1}{\sqrt{10}}, \\
x'=0, \ y'=0, \
a = -\frac{d}{a} = \frac{10}{\sqrt{2}}, \ b = -\frac{e}{c} = \frac{10}{\sqrt{2}}
$$
I have got a system of linear equations
$$
\begin{cases}
x'=a + x \cos{-\alpha} - y \sin{-\alpha} \\
y'=b + x \sin{-\alpha} - y \cos{-\alpha} \\
\end{cases}
\begin{cases}
0 = \frac{10}{\sqrt{2}} + x \frac{1}{\sqrt{10}} + y \frac{3}{\sqrt{10}} \\
0=\frac{10}{\sqrt{2}} - x \frac{3}{\sqrt{10}} + y \frac{1}{\sqrt{10}} \\
\end{cases} 
\implies
x=1, \ y = -2
$$
That is the coordinates of transformed system's origin in the old system is $O(-1,2)$, which is correct
