Calculate principal axes of 2D polygon I would like to calculate the principal axes orientation angle (theta) of a random 2D geometry/polygon and then calculate the minimum enclosing rectangle.
I'm using this formula, based on the inertia moments, for calculating the orientation angle of the principal axes:
tan(2theta) = -2Ixy/(Ix-Iy)
For this example, it works fine for me. I tried different orientations by hand, and finally calculated the orientation angle (-9.7º) using the formula above. When I rotate the polygon this angle, the generated bounding box gives me the smallest area.

However, following the same steps, it does not work in this case.

The result of the formula gives me a value of -23.71º. When I rotate the polygon and generate the bounding box, the area is higher than the original shape.
Note:  I can only get the bounding box aligned to the x and y axes, so what I do is rotate the figure and calculate the new bounding box. Theoretically it is the same as if I could calculate the bounding box at different angles.
 A: The minimal bounding box of a polygon is determined by the polygon's convex hull,
which is the smallest convex polygon that completely covers the original polygon.
If we start with a convex polygon and randomly cut away pieces of it,
only making sure we never cut away one of the original vertices of the polygon, the convex hull (and the minimal bounding box) remains the same, but the axes of inertia depend on how much we cut from what part of the polygon.
Intuitively, the probability that axes of inertia of a random polygon are exactly parallel to the sides of a minimal bounding box should be zero.
But sometimes you get lucky and they're close to parallel, so unless you have a better algorithm you might not discover that you had not computed a minimum.
As you pointed out, what matters is the relative rotation of the polygon and the axes of the bounding box, and we can equally well hold one still and rotate the other. So let's keep looking at rotations of the polygon, and let the bounding box be aligned with the coordinate axes.
An algorithm that can actually find the minimum would look at the vertices of the convex hull that contact the bounding box.
The convex hull is a convex polygon whose vertices are some subset (possibly all) of the vertices of the original polygon.
Suppose the convex hull has $N$ sides.
As we rotate the polygon (and the convex hull along with it)
we will encounter $N$ particular rotation angles, one for each of the sides of the convex hull, at which that side of the convex hull lies along the top edge of the bounding box.
(There are other angles where that side lies along another edge of the bounding box, but they will produce bounding boxes of the same size, so we don't need to worry about them at this time.)
You can deduce the rotation angle for each of these orientations of the polygon by measuring the angle that each side of the convex polygon makes with the $x$ axis.
Try each of these orientations and choose the one with the smallest bounding box.
And that's the entire algorithm.

In case you're interested in what happens at other rotation angles not considered by the algorithm, or wondering why they are not considered:
There are up to $4N$ different rotation angles that lead to one side of the convex hull lying on a side of the bounding box.
(There might be fewer, because one rotation angle could make multiple sides of the convex hull lie on sides of the bounding box.)
For any other orientations of the polygon, there will be only one vertex in contact with each side.
In some of these cases a vertex will be in a corner of the bounding box, contacting two sides at once, so it is possible that the box is defined by as few as two vertices, but never more than four vertices of the convex hull can be on the bounding box at any angle other than the $4N$ (or fewer) orientations where a side lies on an edge of the bounding box.
So let's consider two consecutive rotation angles among that finite set of orientations and consider what happens at angles in between. There will be two vertices that define the height of the box and two that define the width. The height is a function of the length $L_1$ of the side or diagonal between the two vertices that determine the height and the angle $\theta$ by which the polygon has been turned; the side or diagonal between these two vertices then makes an angle $\theta + \alpha$ with the horizontal axis
(where $\alpha$ is a constant determined by the polygon's original shape)
so the height is $L_1 \cos(\theta + \alpha).$
Similarly the side or diagonal of the vertices that determine the width has length $L_2$ and angle $\theta + \beta$ with the vertical axis (where $\beta$ is another constant) so the width is $L_2 \sin(\theta + \beta).$
The area of the polygon within this range of rotation angles is therefore
$$ L_1 L_2 \sin(\theta + \alpha) \sin(\theta + \beta).$$
But
$$
\sin(\theta + \alpha) \sin(\theta + \beta)
= \frac12 (\cos(\alpha - \beta) - \cos(2\theta + \alpha + \beta))
$$
So the area of the bounding box is proportional to
$$ cos(\alpha - \beta) - \cos(2\theta + \alpha + \beta), $$
which is just a simple sinusoidal function,
$-\cos(2\theta + \alpha + \beta),$
offset by the constant $cos(\alpha - \beta).$
Since the minimum of $-\cos(2\theta + \alpha + \beta)$ is $-1$ and the constant
$cos(\alpha - \beta)$ can never be greater than $1,$
the minimum of the area function cannot be positive.
But the area of the bounding box is always positive, so although there is an angle $\theta$ at which the function
$L_1 L_2 \sin(\theta + \alpha) \sin(\theta + \beta)$
reaches a minimum, it cannot be one of the rotation angles in the range where that function actually describes the area of the bounding box.
The minimum for this sinusoidal function must occur at an angle where some other function actually determines the area of the bounding box.
The conclusion is that a minimal bounding box area can occur only at one of the rotation angles at which one of the sides of the convex hull lies on an edge of the bounding box, not at any of the rotation angles in between.
