While trying to come up with an answer to this question without using the theorem used in the existing answer, whose proof is non-trivial, I tried to find $$ \tag{1} \sup_{a a^* + b b^* = 1} | a^2x + aby | $$ for some fixed complex numbers $x, y \in \mathbb{C}$ In the question the conditions $|x| = 1$ and $y \ne 0$ are added but I am interested if there's also a more general solution without those constrains. The task of the question is to prove that $$(1)> |x|.$$
My progress Aiming to use Lagrange multipliers, I expanded the term $a^2x + aby$ in terms of $n_1 := \Re(n)$ and $n_2 := \Im(n)$ for $n \in \{a,b,x,y\}$ to then use that $$\tag{2} |z| = \sqrt{z_1^2 + z_2^2} $$ for all $z \in \mathbb{C}$. I ended up with $$ \Re(a^2x + aby) = a_1 b_1 y_1 + a_1 b_2 y_2 + a_2 b_1 y_2 - a_2 b_1 y_2 + a_2^1 x_1 + a_2^2 x_1 $$ and $$ \Im(a^2 x + aby) = a_1 b_1 y_2 - a_1 b_2 y_1 + a_2 b_1 y_1 + a_2 b_2 y_2 + a_1^2 x_2 + a_2^2 x_2. $$ Plugging this into (2) yields the square root of more than 40 terms. Is there a nice way to factor this or to approach this?
If this were only a real problem, we'd have $$ \sup_{a^2 + b^2 = 1} ax^2 + a b y = \sup_{|a| \le 1} a + a \sqrt{1 - a^2} y $$ Setting the derivative w.r.t to $a$ equal to zero we obtain $$ 0 = \sqrt{1 - a^2} y + 1 - \frac{a^2 y}{\sqrt{1 - a^2}}, $$ giving $$ a^2 = \frac{\pm \sqrt{8 y^2 + 1} - 1}{8y^2} + \frac{1}{2}. $$ Plugging the positive root in gives $$ a + a \sqrt{1 - a^2} y = \left(\sqrt{\frac{\pm \sqrt{8 y^2 + 1} - 1}{8y^2} + \frac{1}{2}}\right)\left(1 + \sqrt{\frac{1}{2} - \frac{\pm \sqrt{8 y^2 + 1} - 1}{8y^2}} \cdot y\right), $$ which looks approximately linear in $y$ when plotted with WolframAlpha. Unfortunately, $\Re\left(\sqrt{\frac{1}{2} - \frac{\pm \sqrt{8 y^2 + 1} - 1}{8y^2}}\right) = 0$ for all $y \in \mathbb{R}$.