Assuming the old rectangle inscribed in the new one, we have the following picture:
Let $\theta$ ($0 \leq \theta \leq \frac{\pi}{2}$) the rotation angle, $w'$ the new width and $h'$ the new height, then we have the following equations:
$$w' = w \cos \theta + h \sin \theta$$
$$h' = w \sin \theta + h \cos \theta$$
The new rectangle is not similar to the old one, except for $h = w$ when both rectangles are in fact squares.
Edit:
Considering $O$ (the center of both rectangles) as the origin of the coordinate system, the points $E$, $F$, $G$, and $H$ can be calculated by the following equations:
$$E=\left(\frac{w}{2}-w \cos^2 \theta,-\frac{h}{2}-w \sin \theta \cos \theta \right)$$
$$F=\left(\frac{w}{2}+h \sin \theta \cos \theta,-\frac{h}{2}+h \sin^2 \theta \right)$$
$$G=\left(-\frac{w}{2}+w \cos^2 \theta,\frac{h}{2}+w \sin \theta \cos \theta \right)$$
$$H=\left(-\frac{w}{2}-h \sin \theta \cos \theta,-\frac{h}{2}+h \cos^2 \theta \right)$$