The unsolved mathematical light beam problem I have the following problem:
Imagine that you have a sphere sitting at the interface of two media(like water and oil). And the position(the heigth) of the interface to the center of the sphere is fixed and known(I drew a picture for two different situations). Now think about this: A beam of light (parallel rays) that enclose an angle alpha to the interface enters (the direction is supposed to be: they first enter the lower medium and then go to the upper one) hits this interface with the sphere. Now the question is: Can we find an analytical expression for the maximum area of the sphere perpendicular to the direction of the rays that is defined by the rays that enter the sphere without going first into the other medium? 
If this is unclear look at the picture: In situation 1 I drew 2 rays (of the infinitely many that enter at this angle) that fulfill the condition that they first hit the sphere without going in the upper medium. Now especially the left one is crucial since if I had chosen one that was even more slightly shifted to the left side, this one would have been in the upper medium, so no longer a reasonable candidate. The right one is not a restriction. The right picture is poorly drawn, since I wanted to have one, where both rays restrict the accessible area. But I think you have the idea now, but what do I mean by area?
I am looking for the biggest area(=PROJECTION OF THE SURFACE AREA of points that fulfill this on a plane)
 perpendicular to the direction of the rays inside the sphere, that consists of rays that enter the sphere and fulfill the property above . So in the first picture this would probably the area going through the center and "somewhat enclosed by the two arrays" and in the second part, this one should be the "imaginary interface" inside the sphere. 

If you have any questions, please do not hesitate to post them
 A: I began writing a comment but it was getting a little long.
I think the problem's definitely manageable, although there are quite a few variables so the calculations won't be pretty (especially as there are several different cases to check). You are essentially working out the surface area of a portion of the sphere which satisfies two inequalities defined by planes. The planes being the interface plane and the plane orthogonal to the vector the light is travelling which goes through the center of the sphere.
In your second example, it turns out that one plane is enough because the plane orthogonal to the light vector is parallel to the interface plane. You will find that you can work out the surface area of a spherical cap using a bit of trig and the known values for the height of the center of the sphere above/below the interface  and the radius of the sphere. Your general example will be the area of the intersection of two spherical caps which I think you may be able to calculate the area of using the inclusion-exclusion principal although I'm not 100% sure of this.
Hopefully someone can edit this if I've said anything incorrect but this should be a good starting point for you.
A: Let us assume that the radius of the sphere is $R$, the height of the center above the separation plane is $h$ (if the center is below the plane, $h$ is negative, and the angle the rays make with the vertical line is $\beta\in(0,\pi/2)$ (it is $\beta=\frac \pi 2-\alpha$ on your picture).
Only the case $|h|<R$ is interesting. Now, 3 possibilities may present itself:
1) $h\le -R\sin\beta$. Then we want the area of the projection of the circle of radius $\sqrt{R^2-h^2}$ to the slanted plane. It is just $A=\pi(R^2-h^2)\cos\beta$.
2) $h> R\sin\beta$. Then the answer is $A=\pi R^2$ (trivially; the border plane doesn't matter for the rays that hit the sphere at all)
3) $-R\sin\beta<h<R\sin\beta$. Then there are two parts: the part of the equator and the part of the cross-cut. The equator projects faithfully and the cross cut with the $\cos\beta$ factor. All we need is to find the area of each. The separating line lies at the distance $H=\frac{|h|}{\sin\beta}$ from the center of the equator disk (for $h>0$, we need the small segment of the equator disk and the large segment of the separation disk; for $h<0$, it is the opposite. The length of that line is $2\sqrt{R^2-H^2}$
Now, if we have a disk of radius $r$ and a chord of length $2\ell$, the (small) segment that is cut off has the area $r^2\arcsin\frac\ell r-\ell\sqrt{r^2-\ell^2}$ (sector minus triangle).
Now, let's bring everything together.
If $h\ge 0$, then we get
$$
A=R^2\arcsin\frac {\sqrt{R^2\sin^2\beta-h^2}}{R\sin\beta}-\frac
{h\sqrt{R^2\sin^2\beta-h^2}}{\sin^2\beta}+
\\
\cos\beta\left[(R^2-h^2)\left(\pi-\arcsin\frac{\sqrt{R^2\sin^2\beta-h^2}}{\sin\beta\sqrt{R^2-h^2}}\right) 
+\frac{h\cos\beta\sqrt{R^2\sin^2\beta-h^2}}{\sin^2\beta}\right]\,.
$$
If $h\le 0$, we get
$$
A=R^2\left(\pi-\arcsin\frac {\sqrt{R^2\sin^2\beta-h^2}}{R\sin\beta}\right)
+\frac{|h|\sqrt{R^2\sin^2\beta-h^2}}{\sin^2\beta}+
\\
\cos\beta\left[(R^2-h^2)\arcsin\frac{\sqrt{R^2\sin^2\beta-h^2}}{\sin\beta\sqrt{R^2-h^2}}
-\frac{|h|\cos\beta\sqrt{ R^2\sin^2\beta-h^2}}{\sin^2\beta} \right]\,.
$$
I agree that this expression may make a nervous person pass out and admit that I might make a stupid typo somewhere when $\LaTeX$-ing it, but I believe I wrote enough for you to be able to make an independent derivation. Enjoy! :)
A: I am working on a solution but need some points clarifying.  You say light is incident from the lower media which is probably the water with oil floating on the top. Am underwater torch could do this. But in the second diagram you seem to have changed the media because the the diagram appears inverted.  Is this the case or is the light incident from the upper side? To make the problem valid we need the same media for both cases.
