Here's a useful piece of terminology: we say that an ideal $I$ is radical if $\sqrt{I}=I$, ie if $x^n\in I$ implies $x\in I$ for all $x$. Note that $\sqrt{I}$ is always radical, for any ideal $I$. (In other words, $\sqrt{\sqrt{I}}=\sqrt{I}$.)
Note that any prime ideal is radical. (Indeed, if $P$ is a prime ideal, you can prove by induction on $n$ that $X^n\in P$ implies $X\in P$ for all $X$, where the base case of $n=2$ follows from the definition of a prime ideal.) Now, you have shown that $\langle Y\rangle\leqslant\sqrt{I}$. On the other hand, $\langle Y\rangle$ is a prime ideal of $R$, since $R/\langle Y\rangle\cong\mathbb{C}[X]$. By the remark above, this means $\langle Y\rangle$ is a radical ideal. Moreover, $I\leqslant\langle Y\rangle$, since $XY\in\langle Y\rangle$ and $Y^2\in\langle Y\rangle$. Thus also $\sqrt{I}\leqslant\sqrt{\langle Y\rangle}=\langle Y\rangle$. (Here we use the general fact that, for any ideal $I,J$, if $I\leqslant J$ then $\sqrt{I}\leqslant\sqrt{J}$.) So $\sqrt{I}=\langle Y\rangle$, and we are done.
In terms of a general approach, here is a piece of advice I would recommend. If you're working with an ideal $I$, and you find an element $a$ such that $a\in\sqrt{I}$, then replace $I$ by $I+\langle a\rangle$ and work from there; often this will simplify the ideal you're working with. You can do this because we have $\sqrt{I+\langle a\rangle}=\sqrt{I}$ for any $a\in\sqrt{I}$ – can you see why this is the case? (Hint: use the two facts that $\sqrt{\sqrt{I}}=\sqrt{I}$ and that $I\leqslant J$ implies $\sqrt{I}\leqslant\sqrt{J}$.) In this case, when $a=Y$, we replace the ideal $I$ with the ideal $\langle Y\rangle$, and then conclude quickly since $\langle Y\rangle$ is prime. So, in a slogan, you don't have to stick with the ideal you start with! You can enlarge it as you find elements in its radical.