Since your answer is tagged precalculus, I'll stick to precalculus methods. Note that one way to solve this would be using Lagrange multipliers from calculus. That calculation is straight-forward, but the numbers get messy. (In some sense, the solution below is a Lagrange multiplier solution, but without the calculus).
In your problem, the first step that I will take is to change variables to $z=\sqrt{2}y$. Then, your problem is to maximize or minimize
$x^2+z^2$ subject to
$x^2-\frac{1}{\sqrt{2}}xz+z^2=1$.
Note that the function that you're maximizing is the square of the distance function from the origin and the constraint equality is an ellipse centered at the origin. The closest point to the origin is on the minor axis and the furthest point from the origin is on the major axis. The problem is that these axes are not axis aligned.
In this case, the major axis is along the line $x=z$ and the minor axis is along the line $x=-z$. This can be seen because the equation is symmetric in $x$ and $z$. Now, when $x=z$, we can substitute $z=x$ into $x^2-\frac{1}{\sqrt{2}}xz+z^2=1$ to get
$x^2-\frac{1}{\sqrt{2}}x^2+x^2=1$.
We can isolate $x^2$ to get
$\left(2-\frac{1}{\sqrt{2}}\right)x^2=1$ or that
$x^2=\frac{\sqrt{2}}{2\sqrt{2}-1}$.
Since $x=z$, the maximum value is then
$\frac{2\sqrt{2}}{2\sqrt{2}-1}$.
On the other hand, along the line $x=-z$, we can substitute into $x^2-\frac{1}{\sqrt{2}}xz+z^2=1$ to get
$x^2+\frac{1}{\sqrt{2}}x^2+x^2=1$.
Once again, we can isolate for $x^2$ to get
$x^2=\frac{\sqrt{2}}{2\sqrt{2}+1}$.
Since $x^2=z^2$, the minimum value is
$\frac{2\sqrt{2}}{2\sqrt{2}+1}$.