How to find the maximum area of a rectangle within a pentagon I am trying to find the maximum area of a rectangle confined within a regular pentagon.
Say the length of the sides of the regular pentagon are $l = 1$ and the length of one of the sides of the rectangle is $x$. As the rectangle rotates I think $x$ will oscillate between $x = l = 1$ and some value $x<1$.
How would one be able to describe this relationship between $x$ and $l$?
 A: This isn't perfect by any means, but here is a crack at this problem. I am going to assume that we can find at least three vertices of the rectangle on three distinct consecutive edges, although I don't know how to prove it, maybe someone else can refute or help.
Given this assumption, consider the following setting in a regular pentagon with side $L$:

where $x+a=L$, $\gamma=3\pi/5$ and $c$ is a side of the rectangle. The unknowns here, or the parameters that we control if you prefer, are $x$ and $\beta$. It's important to note that we can draw the exact same figure for the next side of the rectangle, with $a'=L-b$ and $\beta'=\pi/2-\alpha$. That's how we are going to get a formula for the area $cc'$ of that rectangle as a function of $x$ and $\beta$.
Let us start with the angles:
\begin{align}
  \alpha &= \pi - \beta - \gamma = 2\pi/5 - \beta\\
  \beta' &= \pi/2 - \alpha = \pi/10 + \beta\\
  \alpha' &= 2\pi/5 - \beta' = 3\pi/10 - \beta
\end{align}
Furthermore, the law of cosines gives us:
\begin{align}
  c^2 &= a^2 + b^2 - 2ab\cos\gamma\\
  c'^2 &= a'^2 + b'^2 - 2a'b'\cos\gamma
\end{align}
where we know that:
$$
  a=L-x\qquad a'=L-b
$$
and finally with the law of sines:
$$
  b = \frac{\sin\beta}{\sin\alpha}a = \frac{(L-x)\sin\beta}{\sin(2\pi/5 - \beta)} \qquad
  b' = \frac{\sin\beta'}{\sin\alpha'}a' = \frac{(L-b)\sin(\pi/10 + \beta)}{\sin(3\pi/10 - \beta)}
$$
We now have all the information we need to compute the area $cc'$ as a function of $x$ and $\beta$, with the constraints $0\leq x< L$ and $0\leq\beta< 3\pi/10$. Actually, the least upper-bound on $\beta$ depends on $x$, and will in general be lower than $3\pi/10$ (the lowest one is $\pi/5$ for $x=0$, you can draw it). 
I used the following Matlab program to compute the area of the rectangle for each valid pair $(x,\beta)$, taking $L=1$ (without loss of generality), and setting the area to 0 if any of the sides $a,a',b,b'$ was found negative or greater than 1.
function A = pentarect()

    nx=60; nb=60;

    x = linspace(0,1,nx+1);
    x = x(1:end-1);

    beta = linspace(0,3*pi/10,nb+1);
    beta = beta(1:end-1);

    A = zeros(nb,nx);
    for i = 1:nx
    for j = 1:nb
        A(j,i) = compute_area(x(i),beta(j));
    end
    end

    figure;
    surf(x,beta,A); axis vis3d tight;
    xlabel('x'); ylabel('\beta'); zlabel('Area');
    title('Area as a function of x and \beta with L=1');

end

function area = compute_area(x,beta)

    gamma = 3*pi/5;
    alpha = 2*pi/5 - beta;
    betap = pi/10 - beta;
    alphap = 3*pi/10 - beta;

    a = 1 - x;
    b = a*sin(beta)/sin(alpha);

    ap = 1 - b;
    bp = ap*sin(betap)/sin(alphap);

    if any([b,bp,a,ap] < 0) || any([b,bp,a,ap] > 1)
        area = 0;
    else
        c2 = a^2 + b^2 - 2*a*b*cos(gamma);
        cp2 = ap^2 + bp^2 - 2*ap*bp*cos(gamma);
        area = sqrt(c2*cp2);
    end

end

And here is the resulting plot, showing a clear maximum for $x=\beta=0$:

A: [![enter image description here][1]][1]
[1]: https://i.stack.imgur.com/DGLRw.jpg  I make the assumption, that the largest rectangle within a regular pentagon must be centered on an axis of symmetry of the pentagon, in such a way that all vertices of the rectangle lie on the pentagon's perimeter, as in the accompanying two figures and those of the OP, with two sides of the rectangle parallel to the axis of symmetry and the other two sides parallel to (or one of them coinciding with) the side of the pentagon bisected by that axis (broken line in right-hand figure).  As suggested in the @Claude comment, the method is to derive an equation--actually two equations here--for the rectangle's area and then take derivatives. Under my assumption rotation is not necessary, since an axis of symmetry of the pentagon serves to fix the orientation of the rectangle we seek. **I.**Beginning with the left-hand figure, and letting AB = $1$ and AF = $x$, note that, measured in degrees, $\angle JEF=108$ , $\angle EJF= 54$, and $\angle EFJ = \angle FAK = 18$.  By the law of sines, $$\frac{FJ}{FE}=\frac{sin108}{sin54}$$ Thus$$FJ= (1-x)(\frac{sin108}{sin54})$$and$$FG = 1+2xsin18$$Denoting the area of the rectangle as $y$, we then have $$y = FG*FJ = [1+2xsin18][(1-x)\frac{sin108}{sin54}]$$or finally$$y\approx-.727x^2-.449x+1.176$$Differentiating twice$$y'\approx-1.454x-.449$$and $$y''\approx-1.454$$Since y'' is negative, then setting $y'=0$ it appears $y$ is a maximum for $x\approx-.309$.  But with negative $x$ the rectangle must break out of the pentagon.  Since $0\le x\le 1$ in this figure, evidently $y$ is a maximum for $x=0$, and $y=0$ for $x=1$.  The greatest rectangle has its base = $1$ and height $\approx 1.17557$  **II. **In the right-hand figure, the base of the rectangle lies along a side of the pentagon, $x$ is measured in the opposite direction, and note that $\angle KAF=72$ degrees.  Here $FJ=KJ-KF=(1+x)\frac{sin108}{sin54}-xsin72$, and $FG=1-2xsin18$, so that$$y\approx-.139x^2-.502x+1.176$$ Differentiating twice$$y'\approx-.139x-.502$$and $$y''\approx-.139$$which makes $y$ a maximum for $x\approx-1.806$. But again, in the figure $0 \le x \le1.608...$.  Thus again $y$ is a maximum for $x=0$, and $y=0$ for $x\approx1.608$.  If our assumption is correct, then, the greatest rectangle in a regular pentagon is the one constructed on the side of the pentagon with its other two vertices on the opposite sides.  
