Finding the area remaining after flipping a rectangle inside a rectangle Let $r$ be the inside rectangle of base $b$ and height $h$.
Let $R$ be the outside rectangle of base $B$ and height $H$
The dimensions of $r$ and $R$ are related in the following way:

I want to find the area left when you flip $r$ against the walls of the outside triangle until its back in it's original position, here is a diagram of the area left after a few flips (orange region):
The pink rectangle indicates where the rectangle was before we started flipping it.
 A: The untouched area for $b=h$ (ie, when $B H$ is a square), is $$\text{ area}=b^2 \left(-2 \sqrt{3}-\sqrt{7}+\frac{2 \pi }{3}+7-4 \csc ^{-1}\left(\frac{4}{\sqrt{7}}\right)\right):$$

Here, $B:H=1:1.$ It is likely this is either optimal, or close to optimal.

The area $=0$ for $h\geq\left(\sqrt{7}-2\right) b$ - or, of course, for $b\geq\left(\sqrt{7}-2\right) h:$

Here, $B:H=1:\left(14+3 \sqrt{7}\right)/19$

Some Mathematica code to play with (example implementation: areaBH[2,2] and manipulateBH[2,2]).

Update
The total untouched area can therefore be calculated with:
Area $=$
If $\left(\sqrt{7}-2\right) b<h< \left(\sqrt{3}-1\right),$
\begin{align}{r}
\frac{1}{2} \left(h \left(-\sqrt{4 b^2+3 h^2}-2 \sqrt{b (3 b-4 h)}+4 h\right)+4 \left(b^2+h^2\right) \times\\
\left(\tan ^{-1}\left(\frac{\sqrt{b (3 b-4 h)}}{b+2 h}\right)-\tan ^{-1}\left(\frac{h}{\sqrt{4 b^2+3 h^2}}\right)\right)+2 b h-b \sqrt{b (3 b-4 h)}\right)\\
\end{align}
If $\left(\sqrt{3}-1\right) b<h<b/(\sqrt{3}-1),$
\begin{align}
&\frac{1}{2} \left(-b \left(\sqrt{3 b^2+4 h^2}-6 h\right)-h \left(\sqrt{4 b^2+3 h^2}+2 \left(\sqrt{3}-2\right) h\right)+4 \left(b^2+h^2\right)  \times\\ \left(-\tan ^{-1}\left(\frac{h}{\sqrt{4 b^2+3 h^2}}\right)+\cot ^{-1}\left(\frac{b}{\sqrt{3 b^2+4 h^2}}\right)-\tan ^{-1}\left(\frac{b+\sqrt{3} h}{\sqrt{3} b-h}\right)+\tan ^{-1}\left(\frac{\sqrt{3} h-b}{\sqrt{3} b+h}\right)\right)-2 \left(\sqrt{3}-2\right) b^2\right)\\
\end{align}
If $b/(\sqrt{3}-1)<h<b/(\sqrt{7}-2),$
\begin{align}
b \left(-\frac{1}{2} \sqrt{3 b^2+4 h^2}-\sqrt{h (3 h-4 b)}+h\right)+2 \left(b^2+h^2\right) \left(\cot ^{-1}\left(\frac{b}{\sqrt{3 b^2+4 h^2}}\right)-\tan ^{-1}\left(\frac{2 b+h}{\sqrt{h (3 h-4 b)}}\right)\right)+2 b^2-\frac{1}{2} h \sqrt{h (3 h-4 b)}\\
\end{align}
otherwise, $0.$
Anyhow, the area that the OP asked for can be found with the functions areaBH[b,h] (which is slower) or fBH[b,h] (the formula given above) for any $b$ and $h.$ 
eg fBH[N[100],80] gives $470.616$ almost immediately, whereas fBH[100,80]//FullSimplify gives $-200 \left(82 \sqrt{3}+8 \sqrt{37}+5 \sqrt{139}+\frac{164 \pi }{3}-284-164 \cot ^{-1}\left(\frac{9}{\sqrt{5843-80 \sqrt{5143}}}\right)\right),$ which takes a little longer to calculate.
Note
If $\dfrac{\text{area}}{BH}$ is plotted instead of just $\text{area},$ the maximum value returns to $h=b$ as expected:

With[{b = 1}, Plot[fBH[b, h]/((2 h + b) (2 b + h)), {h, b (Sqrt@7 - 2), 
b/(Sqrt@7 - 2)}, Filling -> Axis, Axes -> False, Frame -> True, 
GridLines -> {{b, b (Sqrt@7 - 2), b/(Sqrt@7 - 2)}, {0, fBH[b, b]/((2 b + b) 
(2 b + b))}}]]

so $b=h$ is indeed optimal, when scaled properly against $BH.$
