Algebraically speaking, there are two ways to solve Equations $(1)$ and $(2)$ simultaneously. One way is $a = b = 1.$ The other is $a=0,$ $b = \frac52.$
You can reject the second solution since it involves a degenerate square.
When the arc is very slightly less than a semicircle, however, you will again get two solutions, both involving three squares of positive dimensions.
In one case $a$ and $b$ will still be nearly $1$; in the other case $a$ will be nearly $0$ and $b$ will be nearly $\frac52.$
If you continue decreasing the angle of the arc, the smaller solution for $a$ increases and the larger solution decreases.
Eventually there is a minimum angle of the arc for which any solution exists, and there is only one solution for that arc.
To find the minimum angle of the arc, note that $b = \frac12(5 - 3a),$ allowing us to reduce the problem to just one variable.
Let the endpoints of the arc be at $\left(\pm\frac52,0\right)$ so that the midpoint of the chord between them is at the origin.
The the center of the the lower left vertex of the pink square is at
$\left(-\frac32a,a+b\right) = \left(-\frac32a,\frac12(5 - a)\right).$
The segment between $\left(\frac52,0\right)$ and this vertex has slope
$-(5-a)/(5+3a)$ and midpoint $\left(\frac14(5-3a),\frac14(5-a)\right)$.
The perpendicular bisector of this segment (which must pass through the center of the circle) has equation
$$y = \frac14(5-a) + \frac{5+3a}{5-a}\left(x - \frac14(5-3a)\right),$$
which (after simplification) has $y$-intercept $-\frac52(a - a^2)/(5-a).$
The minimum angle of arc occurs at the minimum value of the $y$-intercept.
That minimum occurs when $a = 5 - 2 \sqrt5.$
For an arc slightly larger than a semicircle, the number of solutions depends on how you interpret the constraints. If the pink square is permitted to be larger than the purple square, and if the arc must pass through the leftmost point where the two squares touch, there are usually two solutions, one where the purple square is just large enough for its upper left vertex to touch the arc, and another where the purple square's upper left vertex is outside the circle but the lower left vertex of the pink square touches the arc.
If the pink square cannot be larger than the purple square, or if the arc must pass through the lower left vertex of the pink square, then there is only one solution for any arc that is more than a semicircle.
To find the range of possible total areas, assuming the pink and purple squares must both have positive side lengths and that they must exactly occupy one side of the blue square as shown in the figure,
the total area is a function of the edge of the pink square,
\begin{align}
A = f(a) &= a^2 + (a+b)^2 + (2a+b)^2 \\
&= a^2 + \left(\frac12(5-a)\right)^2 + \left(\frac12(5+a)\right)^2 \\
&= \frac12(3a^2 + 25).
\end{align}
Now the range of areas depends on how you specify the problem.
Note that the area is minimized when $a = 0$ (or as $a$ approaches zero, if you do not allow degenerate squares) and is an increasing function of $a$ for $a \geq 0.$
Hence the lower bound of the area will occur at the lower bound of $a.$
For arcs less than a semicircle, if you allow both solutions when two solutions are available, there are arcs yielding all values of $a$ such that $0 < a < 1.$
The lower bound of $a$ is zero, so the lower bound of the area is
$\frac12(3(0^2)+25) = 12.5.$
If you want to restrict the solutions to just one possible solution for each arc, and choose the larger value of $a$ in the case of an arc less than a semicircle, then the minimum area occurs at $a = 5 - 2 \sqrt5,$ and the minimum area is
$10 (8 - 3 \sqrt5) \approx 12.91796.$
If you require the pink square to be no larger than the purple square,
that is, $b \geq 0,$ then the maximum value of $a$ is $\frac53$ and the maximum area is $\frac12\left(3\left(\frac53\right)^2+25\right) = \frac{50}{3} \approx 16.66667.$
If you allow the pink square to be larger than the blue square, then the upper bound of $a$ is $5$ and the upper bound of the area is
$\frac12(3(5^2)+25) = 50.$
This upper bound can be achieved only if you allow the purple square to be degenerate.
If you require the arc to pass through the leftmost point of intersection of the pink and purple squares (rather than the lower left vertex of the pink square), then the arcs that have solutions for $\frac53 < a < 5$ also have solutions for
$1 < a < \frac53,$ so again you have the option to set a rule that selects one of the two possible values. "Smallest value" has the same effect as requiring that the pink square be no larger than the purple square, but "largest value" results in areas between $\frac{50}{3}$ and $50$ being possible but areas between $14$ and $\frac{50}{3}$ not being possible.
If you also choose "largest value" for arcs less than a semicircle but disallow degenerate squares, the range of possible areas is then
$$ \left[10 (8 - 3 \sqrt5), 14\right] \cup \left[\frac{50}{3}, 50\right). $$
But if you choose "smallest value" for all arcs less than a semicircle but disallow degenerate squares, the range of possible areas is
$$ \left(12.5, 10 (8 - 3 \sqrt5)\right] \cup \left[14,\frac{50}{3}\right]. $$
"Largest value" for all arcs with the restriction that the pink square cannot be larger than the purple square results in the range of areas
$$ \left[10 (8 - 3 \sqrt5), \frac{50}{3}\right]. $$
"Largest value" but allowing the pink square to be larger than the purple square, requiring the arc to pass through the lower left vertex of the pink square, and disallowing degenerate squares gives the range of areas
$$ \left[10 (8 - 3 \sqrt5), 50\right). $$
Other reasonable conditions give a range of areas $\left(12.5, \frac{50}{3}\right]$
and yet another set of conditions give $\left(12.5, 50\right].$
In summary, you have several options for how to generalize the problem.
Choose how you think it should be generalized, and the range of possible areas follows.
