How can I calculate the maximum and minimum of this function? 

This is incorrect and I have a feeling I am not doing this correctly. 
 A: I understand your unease. The origin is a saddle point, as the Hessian can say:
$$H(0,0) = \begin{vmatrix} f_{xx} (0,0) & f_{xy} (0,0) \\ f_{xy} (0,0) & f_{yy} (0,0) \end{vmatrix} = \begin{vmatrix} 0 & 1 \\ 1 & 0 \end{vmatrix} = -1.$$
As for nonzero points, I have continued from your system the following way: we can write
$$\lambda = \frac{x}{2y} \text{ and } \lambda = \frac{y}{18x},$$
therefore
$$\frac{x}{y} = \frac{y}{9x} \text{ and } 9x^2 =y^2, \text{ hence } 3x = \pm \, y.$$
Using this on the restriction $9x^2 +y^2 = 5$ yields
$$9x^2 + 9x^2 = 18x^2 = 5 \text{ and } x = \pm \frac{\sqrt{5}}{3 \sqrt{2}}.$$
In total we have four points:
$$
\begin{align}
(x_1, y_1) & = \frac{\sqrt{5}}{\sqrt{2}} \left( \frac{1}{3}, 1 \right), \\
(x_2, y_2) & = \frac{\sqrt{5}}{\sqrt{2}} \left( \frac{1}{3}, -1 \right), \\
(x_3, y_3) & = \frac{\sqrt{5}}{\sqrt{2}} \left( -\frac{1}{3}, -1 \right), \\
(x_4, y_4) & = \frac{\sqrt{5}}{\sqrt{2}} \left( -\frac{1}{3}, 1 \right).
\end{align}
$$
For them find
$$f(x_1,y_1) = f(x_3,y_3) = \frac{5}{6} \text{ and } f(x_2,y_2) = f(x_4,y_4) = - \frac{5}{6},$$
as respectively maximum and minimum.
