Suppose that the upper righthand corner of the rectangle is at the point $\langle x,y\rangle$. Then you know that the area of the rectangle is, as you say, $4xy$, and you know that $$\frac{x^2}{a^2} + \frac{y^2}{b^2} = 1\;.\tag{1}$$
Thinking of the area as a function of $x$, we have $$\frac{dA}{dx}=4x\frac{dy}{dx}+4y\;.$$ Differentiating $(1)$ with respect to $x$, we have
$$\frac{2x}{a^2}+\frac{2y}{b^2}\frac{dy}{dx}=0\;,$$ so $$\frac{dy}{dx}=-\frac{b^2x}{a^2y}\;,$$ and $$\frac{dA}{dx}=4y-\frac{4b^2x^2}{a^2y}\;.$$
Setting this to $0$ and simplifying, we have $y^2=\dfrac{b^2x^2}{a^2}$. From $(1)$ we know that $$y^2=b^2-\frac{b^2x^2}{a^2}\;.$$ Thus, $y^2=b^2-y^2$, $2y^2=b^2$, and $\dfrac{y^2}{b^2}=\dfrac12$. Clearly, then, $\dfrac{x^2}{a^2}=\dfrac12$ as well, and the area is maximized when
$$x=\frac{a}{\sqrt2}=\frac{a\sqrt2}2\quad\text{and}\quad y=\frac{b}{\sqrt2}=\frac{b\sqrt2}2\;.$$