Why can't we convert the area element $dA$ to polar by multiplying the polar expressions for $dx$ and $dy$? Say I have a 2D integral in rectangular coordinates and want to convert to polar. I transform the dx dy starting with:
$$\begin{array}{c}
x = r\cos\theta \\
y = r\sin\theta \\
\end{array}
$$
So ...
$$\begin{array}{c}
dx = \cos\theta\,dr - r \sin\theta\,d\theta \\
dy = \sin\theta\,dr + r \cos\theta\,d\theta \\
\end{array}
$$
Then expanding out the multiplication ...
$$\begin{align}
dx\,dy &= \cos\theta\,\sin\theta\,dr^2 - r^2\,\cos\theta\,\sin\theta\,d\theta^2
         + r\,\cos^2\theta\,dr\,d\theta - r\,\sin^2\,\theta\,dr\,d\theta \\
&= r\,dr\,d\theta + \cos\theta\,\sin\theta\,(dr^2 - r^2\,d\theta^2) - 2\,r\,\sin^2\theta\,dr\,d\theta \hspace{3em}\text{... uh oh!}
\end{align}
$$
So there's $r\,dr\,d\theta$ but with some extra terms. Why doesn't this naive and direct approach work out?
 A: Intro
There are two main types of ways to think about things like $\mathrm{d}x\mathrm{d}y$ in multivariable calculus, and we often switch between them depending on the context. (This clarification was inspired in part by Terry Tao's preprint on “differential forms and integration”.) $\mathrm{d}x$ can either act kind of like a number, or act kind of like a vector.
For the “number” interpretation, there are things like limit arguments or infinitesimals in nonstandard analysis to make things rigorous. For the “vector” interpretation, there are things like “differential forms” or “geometric calculus” to make things rigorous. But I'm going to gloss over those details because there are many ways to make things formal, and the exact choices don't affect the intuition here.
Numbers
One way to think about things is that $\mathrm{d}x$ and $\mathrm{d}y$ are in some way like tiny numbers representing the width and length of a tiny rectangle, so that $\mathrm{d}x\mathrm{d}y$ is the area of a tiny rectangle. Then when we write something like $\iint f\left(x,y\right)\,\mathrm{d}x\mathrm{d}y$ or $\iint g\left(r,\theta\right)\,\mathrm{d}r\mathrm{d}\theta$, we just add up the signed volumes (in case $f$ or $g$ is negative) of thin rectangular prisms with cross-sectional area represented by $\mathrm{d}x\mathrm{d}y$ or $\mathrm{d}r\mathrm{d}\theta$.
Under this interpretation, $\mathrm{d}x=\cos\theta\mathrm{d}r-r\sin\theta\mathrm{d}\theta$ doesn't make too much sense. For example, if $\theta=\pi/2$, then we would have $\mathrm{d}x=-r\mathrm{d}\theta$, so that $\mathrm{d}\theta$ and $\mathrm{d}x$ couldn't both represent positive lengths. But we can still understand the relationship between the areas $\mathrm{d}x\mathrm{d}y$ and $\mathrm{d}r\mathrm{d}\theta$ with arguments like the geometric one in this answer by Mike Spivey.
Vectors
The other way to think about things is that $\mathrm{d}x$ and $\mathrm{d}y$ are in some way like tiny vectors whose direction we care about, and this leads to a slightly different discussion. To emphasize this vector idea, I will use some nonstandard notation. Let's write $\overrightarrow{\mathrm{d}x}=\left\langle \Delta x,0,0\right\rangle$  for some positive $\Delta x$, and $\overrightarrow{\mathrm{d}y}=\left\langle 0,\Delta y,0\right\rangle$  for some positive $\Delta y$. So $\overrightarrow{\mathrm{d}x}$ points to the right in the $xy$-plane and $\overrightarrow{\mathrm{d}y}$ points “up” in the $xy$-plane. Then the area of the little rectangle they make is $\left\Vert \overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}\right\Vert =\Delta x\Delta y$.
However, now that we have vectors, we could choose to care about the orientation. When we think about a usual integral like $\int_{\left[a,b\right]}f\left(x\right)\,\mathrm{d}x$ when $f$ is negative, we decide to count that area in a negative way. Similarly, we could count the “signed area” of the oriented rectangle with sides $\overrightarrow{\mathrm{d}x},\overrightarrow{\mathrm{d}y}$ as the positive value $\Delta x\Delta y$ (which is just the $z$-component of $\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}$). But the signed area of the oriented rectangle with sides $\overrightarrow{\mathrm{d}y},\overrightarrow{\mathrm{d}x}$ would be the negative value $-\Delta x\Delta y$ (which is just the $z$-component of $\overrightarrow{\mathrm{d}y}\times\overrightarrow{\mathrm{d}x}$).
Then when we write something like $\iint f\left(x,y\right)\,\mathrm{d}x\mathrm{d}y$ or $\iint g\left(r,\theta\right)\,\mathrm{d}r\mathrm{d}\theta$, we could (if we chose) care about the order of things, with something like an implied $\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}$ or $\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}$ in our heads, if not always our notation.
When using things like vectors, then something like $\overrightarrow{\mathrm{d}x}=\cos\theta\overrightarrow{\mathrm{d}r}-r\sin\theta\overrightarrow{\mathrm{d}\theta}$ makes a lot of sense. $\overrightarrow{\mathrm{d}r}$ points away from the origin, and $\overrightarrow{\mathrm{d}\theta}$ points perpendicularly to $\overrightarrow{\mathrm{d}r}$ in the counterclockwise way (so that $\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}$ points in the same positive-$z$ direction as $\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}$).
Finally, we can correct your calculation:
$$\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}=\left(\cos\theta\overrightarrow{\mathrm{d}r}-r\sin\theta\overrightarrow{\mathrm{d}\theta}\right)\times\left(\sin\theta\overrightarrow{\mathrm{d}r}+r\cos\theta\overrightarrow{\mathrm{d}\theta}\right)$$
$$=\cos\theta\sin\theta\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}r}-r^{2}\sin\theta\cos\theta\overrightarrow{\mathrm{d}\theta}\times\overrightarrow{\mathrm{d}\theta}+r\cos^{2}\theta\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}-r\sin^{2}\theta\overrightarrow{\mathrm{d}\theta}\times\overrightarrow{\mathrm{d}r}$$
$$=\overrightarrow{0}-\overrightarrow{0}+r\cos^{2}\theta\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}-r\sin^{2}\theta\left(-\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}\right)=\boxed{r\,\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}}$$
