Say the side legnths are $s_1, s_2$. Then, one equation is $4 s_1 + 4 s_2 = 40$ (by the length of the wire).
The area enclosed is $s_1^2+s_2^2$. Solve the equation you got for the length of the wire for either $s_1$ or $s_2$, substitute it into the expression for the area enclosed. You will get a quadratic, which you can minimize for side lengths between $0$ and $10$ (so there is enough wire, and side lengths can't be negative since $s_1 \geq 0, s_2 \geq 0$). The minimum will occur at either the vertex of the parabola defined at the quadratic, at side = 0 or side = 10. Then compute the other side, done.