Shape of union of all sector bounded ellipses with additional foci constraints Consider an ellipse constrained inside a sector of a circle $y^2+x^2=\beta^2$ and two lines $y=x\tan\alpha$ and $y=-x\tan\alpha$, where $\beta\geq1$ and $\alpha\in[0,\pi/2)$. Let the equation of this ellipse to be
$$\frac{(x-k)^2}{a^2}+\frac{y^2}{b^2}=1.$$
where we assume that $b> a\geq0$. An example of such ellipse is given in LHS of figure 1. The union of all such ellipses is shown in RHS of figure 1, i.e., the whole sector can be filled by such ellipses by adjusting $a,b$ and $k$.

Now consider one more additional constraint: the foci of an ellipse must lie in red shaded region as shown in LHS of figure 2, i.e., the foci must lie in a sector between two circles with centers at origin and radiuses $\beta$ and $\beta^{-1}$, and two lines $y=x\tan\alpha$ and $y=-x\tan\alpha$. I want to find the shape of union of all such ellipses, that might look like something like RHS of figure 2.

I am looking for any suggestions on how to tackle this problem (i.e., finding exact shape of union of such ellipses in terms of $\beta$ and $\alpha$). I couldn't find similar problem in literature.
Or is it possible to find at least the line that is perpendicular to $x$ axis and tangents this union from the left?
 A: Intuitively, your cartoon seems right. You should get everything near the big circle, and lose stuff inside the inner circle. So let's look at that left boundary. We will make two simplifying assumptions that should be true of ellipses touching the boundary:

*

*The foci lie on the inner circle.

*The ellipse is tangent to the outer lines.

The foci are positioned at $$\left(k, \sqrt{b^2-a^2}\right),\left(k, -\sqrt{b^2-a^2}\right).$$ In order to satisfy the condition that the foci lie on the inner circle, you have:
$$\beta^{-2} = k^2 + b^2 - a^2 $$
The condition of the ellipse being tangent to the line is
$$\dfrac{(x - k)^2}{a^2} + \dfrac{x^2 \tan^2 \alpha}{b^2} = 1, \text{ and } \dfrac{dy}{dx} = -\dfrac{b^2(x-k)}{a^2x\tan\alpha} = \tan \alpha.$$
Eliminating the $x$ gives the constraint $$\tan^2\alpha = \dfrac{b^2}{k^2 - a^2}.$$
The quantity $b$ is actually determined by $$b = \sqrt{\dfrac{\beta^{-2}\tan^2\alpha}{\tan^2\alpha+1}} = \dfrac{\beta^{-1}\tan \alpha}{\sec \alpha} = \beta^{-1}\sin \alpha.$$
Therefore, fixing $k$ determines $a$ as well, via $a^2 = k^2 + \beta^{-2}(\sin^2\alpha-1)= k^2 - \beta^{-2}\cos^2\alpha.$ So you have a family of curves indexed by the parameter $k$. You want to compute the envelope of that family. The Wikipedia page on envelopes  should get you the rest of the way.
