Area of the shadow of a regular polygon inscribed in a sphere. Consider the situation given below:
Let a regular polygon be inscribed in a sphere such that its circumcentre is at a distance $r$ from the centre of the sphere of radius $R$. A point source of light is kept at the centre of the sphere. How can we calculate the area of the shadow made on the surface of the sphere.
I tried to use the relation: $ \Omega = \frac{S}{R^2} $
But of course that is the case when a circle would be inscribed. So can I somehow relate it for any general polygon?
 A: Let $\theta = \cos^{-1}\frac{r}{R}$ and $\phi = \frac{2\pi}{n}$ where
$n$ is the number of sides of the regular polygon.
Choose the coordinate system so that the vertices of the regular polygon are located at
$$\vec{v}_k = (R\sin\theta\cos(k\phi), R\sin\theta\sin(k\phi), R\cos\theta) \quad\text{ for } k = 0, 1, \ldots, n - 1.$$
Let $\hat{v}_k = \frac{\vec{v}_k}{|\vec{v}_k|}$ be the corresponding unit vectors
and $\hat{z} = (1,0,0)$.
Recall following formula by Oosterom and Strackee on solid angle,

Given any 3 unit vectors, $\vec{a}, \vec{b}, \vec{c}$, the solid angle $\Omega$ spanned by these 3 vectors satisfy
  $$\tan\frac{\Omega}{2} = \frac{ \vec{a} \cdot (\vec{b} \times \vec{c}) }{1 + \vec{a}\cdot\vec{b} + \vec{b}\cdot\vec{c} + \vec{c}\cdot\vec{a}}$$

Apply this to the unit vectors $\hat{v}_0, \hat{v}_1$ and $\hat{z}$, we find the corresponding solid angle $\Omega$ satisfy
$$\tan\frac{\Omega}{2} = \frac{\sin^2\theta\sin\phi}{1 + 2\cos\theta + \cos^2\theta + \sin^2\theta\cos\phi}
= \frac{\sin\phi}{
\frac{1+\cos\theta}{1-\cos\theta} + \cos\phi}
= \frac{\sin\phi}{\frac{R+r}{R-r}+\cos\phi}
$$
So the desired area is
$$\verb/Area/ = nR^2\Omega = 2nR^2\tan^{-1}\left[\frac{\sin\left(\frac{2\pi}{n}\right)}{\frac{R+r}{R-r}+\cos\left(\frac{2\pi}{n}\right)}\right]$$
Update
Let $s = \sin\frac{\pi}{n}$, $c = \cos\frac{\pi}{n}$ and $D = \frac{R+r}{R-r}$, we 
can simplify above expression as
$$\begin{align}\tan^{-1}\left[\frac{2sc}{D+c^2-s^2}\right]
&= \frac{\pi}{n} - \tan^{-1}\left[\frac{\frac{s}{c} - \frac{2sc}{D+c^2-s^2}}{1 + \frac{s}{c}\frac{2sc}{D + c^2 - s^2}}\right]
= \frac{\pi}{n} - \tan^{-1}\left[\frac{s(D-1)}{c(D+1)}\right]\\
&= \frac{\pi}{n} - \tan^{-1}\left[\frac{sr}{cR}\right]
\end{align}$$
So a simpler version of the area is
$$\verb/Area/ = 2R^2\left(\pi - n\tan^{-1}\left(\frac{r}{R}\tan\frac{\pi}{n}\right)\right)$$
It is easy to see this equals to the expression in Harish Chandra Rajpoot's answer.
A: Notice, let $a$ be each side of n-sided regular polygon inscribed in a sphere of radius $R$, at a normal distance $r$ from its center. Then the circumscribed radius say $r_c$ of the regular polygon is given by general formula as follows $$r_c=\color{blue}{\frac{a}{2}\csc\left(\frac{\pi}{n}\right)}$$
Drop a perpendicular from center of sphere to the center of regular polygon & join the center of sphere to the vertex of polygon to obtain a right triangle & then apply Pythagorean theorem, to get circumscribed radius of regular polygon,
$$r_c=\sqrt{R^2-r^2}$$
hence, equating the values of $r_c$,
$$\frac{a}{2}\csc\left(\frac{\pi}{n}\right)=\sqrt{R^2-r^2}$$
$$a=2\sin\left(\frac{\pi}{n}\right)\sqrt{R^2-r^2}$$
Now, the solid angle subtended by any regular n-polygon, with each side $a$,  at any point lying at a normal distance $h$  from the center, is given by the general formula (see for detailed explaination & see eq(7) for expression ) as follows
$$\bbox[5px, border:2px solid #C0A000]{\color{red}{\Omega=2\pi-2n\sin^{-1}\left(\frac{2h\sin\left(\frac{\pi}{n}\right)}{\sqrt{4h^2+a^2\cot^2\left(\frac{\pi}{n}\right)}}\right)}}$$
Now, setting the corresponding values, normal distance of regular polygon from the center of sphere, $h=r$, each side of regular polygon, $a=2\sin\left(\frac{\pi}{n}\right)\sqrt{R^2-r^2}$, one should get solid angle subtended by the regular polygon inscribed in the sphere, at the center of sphere, $$\Omega=2\pi-2n\sin^{-1}\left(\frac{2r\sin\left(\frac{\pi}{n}\right)}{\sqrt{4r^2+4\sin^2\left(\frac{\pi}{n}\right)(R^2-r^2)\cot^2\left(\frac{\pi}{n}\right)}}\right)$$
$$=2\pi-2n\sin^{-1}\left(\frac{1}{\sqrt{1+\left(\frac{R}{r}\right)^2\cot^2\left(\frac{\pi}{n}\right)}}\right)$$
since, the solid angle $\Omega$, subtended at the center of sphere, by the regular polygon & its shadow, is equal hence the area of the shadow of regular polygon made on the surface of the sphere is $$S=\Omega R^2$$ $$S=\left(2\pi-2n\sin^{-1}\left(\frac{1}{\sqrt{1+\left(\frac{R}{r}\right)^2\cot^2\left(\frac{\pi}{n}\right)}}\right)\right)R^2$$
$$\bbox[5pt, border:2.5pt solid #FF0000]{\color{blue}{S=2R^2\left(\pi-n\sin^{-1}\left(\frac{1}{\sqrt{1+\left(\frac{R}{r}\right)^2\cot^2\left(\frac{\pi}{n}\right)}}\right)\right)}}$$
$\forall\ \ \ \ \color{red}{0<r<R}$ & $\color{red}{n\ge 3,\ \ n\in \mathbb{N}}$
A: Here is a solution that uses spherical trigonometry. There's a fair amount of geometry, but not too many calculations. All the formulas needed can be found in the Wikipedia article on spherical trigonometry.
Let $O$ be the centre of the sphere, and let $A$ be the centre of the $n$-gon. Divide the polygon into $n$ isosceles triangles in the obvious way (with vertex at $A$), and then divide each of those in half by its axis of symmetry. So the desired area will be $2n$ times the area of the shadow of one of these little right triangles.
Let $ABC$ be one of the little triangles, where $B$ is a vertex of the polygon, and $C$ is the midpoint of one of the sides adjacent to $B$. $ABC$ is a right triangle with right angle at $C$. We are going to look at the trihedral angle formed by the rays $OA$, $OB$ and $OC$. This will correspond to a triangle $A'BC'$ on the surface of the sphere.
$OA$ is perpendicular to the plane $ABC$ of the polygon. Therefore the dihedral angle along edge $OA$ between faces $OAB$ and $OAC$ is equal to $\angle BAC = \pi/n$.
Next we check that the dihedral angle along edge $OC$ is a right angle, i.e., that planes $OCB$ and $OCA$ are perpendicular. Again, since $OA$ is perpendicular to plane $ABC$, it is perpendicular to $BC$. But $BC$ is also perpendicular to $AC$. Since $BC$ is perpendicular to two intersecting lines $AC$ and $OA$, it is perpendicular to their plane $OAC$. Since $OBC$ contains a line $BC$ perpendicular to $OAC$, the plane $OBC$ is itself perpendicular to $OAC$. 
Finally, let $\gamma$ be the angle of face $OAB$, that is, the angle between rays $OA$ and $OB$. $OAB$ is a right triangle with right angle at $A$, and we have $OA = r$, $OB = R$. Therefore $\cos \gamma = r/R$.
In summary, the spherical triangle $A'BC'$ has $C = \pi/2$, $A = \pi/n$ and (opposite $C$) $\cos \gamma = r/R$. Its area is $R^2 E$ where the spherical excess is $E = A + B + C - \pi$, so we just need to find $B$. For this, we apply the relation $\cos \gamma = \cot A \cot B$ valid in right spherical triangles. Therefore 
$$\tan B = \frac{\cot A}{\cos \gamma} = \frac{R}{r\tan \pi/n}.$$
The spherical excess of each little triangle is therefore 
$$E = \pi/n + \arctan\left(\frac{R}{r\tan \pi/n}\right) + \pi/2 - \pi.$$
The total area of the shadow is
$$S = 2n R^2 E = \left[2n \arctan\left(\frac{R}{r\tan \pi/n}\right) - (n-2)\pi\right]R^2.$$
Edit. Using the relation $\arctan x = \pi/2 - \arctan(1/x)$, we see that this is the same answer Achille Hui has.
