You definitely had a correct approach for solving the problem; the only mistake you made is in the tedious algebra of the cross product. I found that
\begin{align}
dS = r \sqrt{1 + r^2 \cos^2 \theta } \, dr \, d\theta
\end{align}
So, the integral you need to compute is
\begin{align}
I &= \int_0^{\pi/2} \int_0^1 (r \cos \theta) \cdot r \sqrt{1 + r^2 \cos^2 \theta } \, dr \, d\theta \\
&= \int_0^{\pi/2} \int_0^1 r^2 \cos \theta \sqrt{1 + r^2 \cos^2 \theta } \, dr \, d\theta. \tag{$\ddot{\smile}$}
\end{align}
The justification for the book writing $dS = \sqrt{1 + x^2} \, dx \, dy$ is pretty much the same justification as to how you computed $dS$ using polar coordinates. In general, whenever you can parametrize a surface $S$, you can always compute $dS$ in terms of those coordinates. In the special case where the surface $S$ can be represented as the graph of a function $\zeta$, we get a nice result. Here's what I mean: define the set
\begin{align}
U := \{(x,y) \in \Bbb{R}^2 | \, \, \text{$x,y \geq 0$ and $x^2 + y^2 \leq 1$} \}
\end{align}
and define the function $\zeta: U \to \Bbb{R}$ by
\begin{align}
\zeta(x,y) = \dfrac{x^2}{2}.
\end{align}
Then, the surface $S$ is precisely the graph of $\zeta$:
\begin{align}
S = \text{graph}(\zeta) = \left\{\big( x,y,\zeta(x,y) \big) | \, (x,y) \in U \right\}
\end{align}
Since $S$ is the graph of a function, it admits a very simple parametrization, namely $\alpha : U \to \Bbb{R}^3$ defined by
\begin{align}
\alpha(x,y) := \big(x,y, \zeta(x,y) \big).
\end{align}
Now, to compute the surface element, you apply an identical formula:
\begin{align}
dS = \left| \dfrac{\partial \alpha}{\partial x} \times \ \dfrac{\partial \alpha}{\partial y}\right| \, dx \, dy.
\end{align}
If you carry out this computation, you'll find that it simplifies to
\begin{align}
dS = \sqrt{1 + \left(\dfrac{\partial \zeta}{\partial x} \right)^2 + \left(\dfrac{\partial \zeta}{\partial y} \right)^2} \, dx \, dy \tag{$*$}
\end{align}
Note that $(*)$ holds for any (nice enough) function $\zeta$. In your special case, $\zeta(x,y) = x^2/2$ finally simplifies everything to $dS = \sqrt{1+ x^2} \, dx \, dy$.
So, what the book did was to recognize that $S$ is the graph of a certain function $\zeta$, hence it can be parametrized easily using $\alpha$, and by using the same cross product argument, you compute $dS$. Now, formula $(*)$ is very helpful in computations, so you should commit it to memory (after understanding of course).
After doing this calculation of $dS$, the integral you need to calculate is
\begin{align}
I = \iint_U x \sqrt{1 + x^2} \, dx \, dy \tag{$\ddot{\smile} \ddot{\smile}$}
\end{align}
which you can try to solve using Fubini's theorem.
By the way, you should notice that if you now change to polar coordinates in $(\ddot{\smile} \ddot{\smile})$, then you recover the integral in $(\ddot{\smile})$. To answer your last question about effectiveness, I think in this particular example, sticking to cartesian coordinates might be a better option, because the integral in $(\ddot{\smile} \ddot{\smile})$ seems easier to solve. By applying Fubini's theorem, I managed (with some difficulty) to calculate it, and I found the answer to be $\pi/8$ (I may have made a mistake along the way)... but I still have no idea how to solve it in polar coordinates.
