With regard to topic/question New size of a rotated-then-cropped rectangle:
The answer by Isaac,
the maximum area is $b^2\csc\alpha\sec\alpha$ when $x=0.5b\csc\alpha = 0.5b/\sin\alpha$
seems to work for angles near $45$ degrees. However, its range of validity seems to be aspect ratio dependent. For $w/h$ aspect ratio $>> a/b$, it works for a large range of angles, but for aspect ratio $= 1$ it only works at $45$ degrees.
This is easily seen by using $\alpha$ nearly zero (very little rotation), then $\sin\alpha$ becomes nearly zero and $x$ becomes very large, certainly much larger than $a$. Whereas the correct solution will be nearly the same size as the slightly rotated rectangle.
The solution, $x=0.5b/\sin\alpha$, leads to $y=0.5b/\cos\alpha$. So when $x=y$ (upper right corner of inner rectangle) and $a=b$ (aspect ratio$=1$), we have $\sin\alpha = \cos\alpha$, which is valid (in the first quadrant) only at $\alpha=45$ degrees.
In particular, the limiting angle would appear to occur when $x=a$, so that $\sin\alpha > 0.5b/a$.
It appears that within the valid range of this solution, only $2$ diagonal vertices of the inner rectangle touch the sides of the rotated rectangle. But in the invalid range, all four vertices of the inner rectangle touch an appropriate side of the rotated rectangle.
I have tried solving this with lagrange multiplers and get the same answer. But I don't know how to put in an inequality constraint that $x\leq a$.
I am wondering if this can be solved closed form with a more general solution that gives a solution for any angle and any aspect ratio. Or at least another solution that works where this solution fails. So far I have been unsuccessful.
A more general and similar problem has been solved by testing multiple conditions, but is not closed form. see http://cgm.cs.mcgill.ca/~athens/cs507/Projects/2003/DanielSud/
I would appreciate hearing from the Mathematics experts on this.