Trigonometric diophantine equation $8\sin^2\left(\frac{(k+1)\pi}{n}\right)=n\sin\left(\frac{2\pi}{n}\right)$ Background. I thought up the problem of finding a regular $n$-sided polygon that has a diagonal with length $d_k$ such that the area of the polygon equals ${d_k}^2$. (Let $d_k$ denote the length of a diagonal that skips $k$ vertices, so that $d_0$ is the length of a side.) By doing some trigonometry I got the equation

$$8\sin^2\left(\frac{(k+1)\pi}{n}\right)=n\sin\left(\frac{2\pi}{n}\right).$$

I have checked some values of $n$ and it seems that $(4,0)$ and $(12,3)$ are the only (primitive and non-trivial) possible values for $(n,k)$.

Question. Are there other solutions?

One approach could be rewriting the equation using $\sin(x)=\frac i2(e^{-ix}-e^{ix})$. This gives $$4(\zeta_n^{k+1}+\zeta_n^{-k-1})-in(\zeta_n+\zeta_n^{-1})=8,$$ where $\zeta_n=e^{2\pi i/n}$.


Update. Except for $n=4$, we have $n=4p$ with $p\equiv3\pmod4$ prime. Furthermore, $\gcd(n,k+1)=4$.

See my answer below. Thanks to user dinoboy for the suggested technique.
 A: (4,0) and (12,3) are the only "primitive and non-trivial" solutions $(n,k)$ with $n\leq 10^5$.
Considering the symmetries that are present both in your geometry question and in the resulting equation, let a "primitive and non-trivial" solution be a pair $(n,k)$ with $n,k\in \mathbb{Z}$, $n\geq 3$, and $0\leq k \leq n/2 - 1$, satisfying$$8\sin^2\left(\frac{(k+1)\pi}{n}\right)=n\sin\left(\frac{2\pi}{n}\right).$$
The fact that $k$ can be restricted to a finite interval for each $n$ opens the problem up for a direct numerical search. Here's my implementation in PARI/GP:
N_MIN=3;
N_MAX=100000;
N_BLOCKSIZE=100;
TOLERANCE=1e-20;

\\ We express the problem in terms of m,n, where m=k+1.
\\ We only call the (slow) sin function once for each computation of the LHS.
lhs(m,n)={sqrt_lhs=sin(m*Pi/n); sqrt_lhs*sqrt_lhs;}
rhs(n)=sin(2*Pi/n)*n/8;

\\ Check m,n for |LHS-RHS| < TOLERANCE
for(n=N_MIN,N_MAX,cur_rhs=rhs(n); if(n%N_BLOCKSIZE==0,printf("# Done checking all n<%u\n", n)); for(m=1,n/2,{
    cur_lhs=lhs(m,n);
    if(abs(cur_lhs-cur_rhs)<TOLERANCE, printf("%u\t%u\t%e\n", n, m-1, cur_lhs-cur_rhs))
    }));
quit


Note: If I'm not mistaken, this is guaranteed to find all solutions $(m,n)$ with $n<n_\max = 10^5$, even though floating point math is used (assuming there are no bugs in PARI/GP or my computer). The point is that PARI/GP claims a default precision of 38 decimal digits, while the tolerance for root-finding in the above code is equivalence to 20 decimal digits. If you calculate the error propagation assuming 38-digit precision for the sin functions, the maximum absolute error after the few arithmetic operations per $n,m$ pair is still on the order of $10^{-38}$. The code can in principle give false positives if the RHS and LHS aren't equal but happen to be within the tolerance of one another, but it can't give false negatives. (In fact for these parameters there were no false positives, either.)

Update: Here is an implementation in C using 128-bit floating point arithmetic:
#include <stdio.h>
#include <quadmath.h>

// To compile: gcc -O3 -std=c99 -lquadmath trig.c -o trig

#define M_2PIq 6.2831853071795864769252867665590058q

#define N_MIN 3
#define N_MAX 2000
#define N_BLOCKSIZE 100
#define TOLERANCE 1e-20q
#define DIGITS 35

int main(){

  // We express the problem in terms of m,n, where m=k+1.
  for(unsigned n=N_MIN; n<=N_MAX; n++){

    // Print out a status update each N_BLOCKSIZE iterations.
    if(n%N_BLOCKSIZE==0) printf("# Done checking all n<%u\n", n);

    // Compute the RHS only once per n
    __float128 cur_rhs = sinq(M_2PIq/n)*n/8.0q;
    for(unsigned m=1; 2*m<=n; m++){

      // We only call the (slow) sin function once for each computation of the LHS.
      __float128 cur_lhs=sinq(m*M_PIq/n);
      cur_lhs*=cur_lhs;

      // Print out (n,k) if a solution is found
      if( fabsq(cur_lhs-cur_rhs) < TOLERANCE ){
        char str_difference[DIGITS];
        quadmath_snprintf(str_difference, DIGITS, "%Qe", fabsq(cur_lhs-cur_rhs));
        printf("%u\t%u\t%s\n", n, m-1, str_difference); 
      }
    }
  }

}

A: (With thanks to Dinoboy's link to the similar question For which $(n,m,l)$ does $L_n^2+L_m^2=L_l^2$ hold?.)
The only solutions $(n, k)$ (with $0 \leq k+1 \leq n/2$) are $(4, 0), (12, 3)$.
Step 1. $4 \mid n$.
Proof. Write $\zeta_n = e^{2\pi i / n}$. Using the double-angle formula $\sin^2 x = (1-\cos(2x))/2$ we obtain
$$2\cos\left(\frac{2\pi(k+1)}{n}\right)+\frac n2 \sin\left(\frac{2\pi}{n}\right)=2 \,.$$
We have that $2 \cos\left(\frac{2\pi(k+1)}{n}\right)$ and $2 \sin\left(\frac{2\pi}{n}\right)$ are algebraic integers, and $\sin\left(\frac{2\pi}{n}\right)$ is not when $n > 2$: indeed, its norm in $\mathbb Q(\zeta_n)/\mathbb Q$ has absolute value
$$0 < \prod_{(j, n) = 1} \frac{|\zeta_n^j - \zeta_n^{-j}|}{2} < 1 \, ,$$
so it cannot be an integer. Because $\frac n2 \sin\left(\frac{2\pi}{n}\right)$ must be an algebraic integer, we obtain that $4 \mid n$.
Step 2. Taking traces.
Using that $i = \zeta_4 = \zeta_n^{n/4}$, we obtain
$$(\zeta_n^{k+1} + \zeta_n^{-k-1}) - \frac{n}{4} (\zeta_n^{n/4 + 1} - \zeta_n^{n/4-1}) = 2 \,.$$
From Derivatives of the nth cyclotomic polynomial, we have $\operatorname{Tr}_{\mathbb Q(\zeta_m)/\mathbb Q}(\zeta_m) = \mu(m)$. Note also that $[\mathbb Q(\zeta_m) : \mathbb Q] = \varphi(m)$ and that $\mathbb Q(\zeta_m^\ell) = \mathbb Q(\zeta_{m / \gcd(m, \ell)})$. Thus for all $\ell \in \mathbb Z$,
$$\operatorname{Tr}_{\mathbb Q(\zeta_n)/\mathbb Q}(\zeta_n^\ell) = \frac{[\mathbb Q(\zeta_n) : \mathbb Q]}{[\mathbb Q(\zeta_{n / \gcd(n, \ell)}) : \mathbb Q]} \mu(n / \gcd(n, \ell)) = \varphi(n) \frac{\mu(n / \gcd(n, \ell))}{\varphi(n / \gcd(n, \ell))} \,.$$
Let $d=(n,k+1)$. Taking $\operatorname{Tr}_{\mathbb Q(\zeta_n)/\mathbb Q}$ and dividing by $\varphi(n)$, we obtain
$$\bbox[5px, border:2px solid black]{2\frac{\mu(n/d)}{\varphi(n/d)}-\frac n4 \frac{\mu(n/(n,\frac n4+1))}{\varphi((n/(n,\frac n4+1))}+ \frac n4 \frac{\mu(n/(n,\frac n 4-1))}{\varphi((n/(n,\frac n 4-1))}=2 \,.}$$
Step 4. $n = 4$ or $n = 4p$ with $p$ prime and $p \equiv 3 \pmod 4$, and $d = 4$.
Proof. We work with this last equation. Casework:


*

*If $8\mid n$, the last two terms in the LHS are $0$ due to $\mu$. So $\mu(n/d)=\varphi(n/d)=1$, so $d=n$ and $n\mid k+1$. The original equation becomes $n\sin(2\pi/n)=0$, impossible.

*If $n\equiv4\pmod{16}$, $(n,\frac n4+1)=2$ and $(n,\frac n4-1)=4$, and vice versa if $n\equiv12\pmod{16}$. Either way, $\varphi((n/(n,\frac n4+1))=\varphi((n/(n,\frac n4-1))$, the $\mu$'s have opposite sign and we get
$$\frac{\mu(n/d)}{\varphi(n/d)}\pm\,\frac n4\cdot\frac{\mu(n/4)}{\varphi(n/4)}=1.$$
where the sign is $+$ when $n \equiv 4 \pmod{16}$ and $-$ when $n \equiv 12 \pmod{16}$.


*

*If $\mu(n/d)=0$ then $n/4=\varphi(n/4)$, so $n=4$. Either by the geometric interpretation or from the starting equation we get $k=0$ or $k=3$. We may now assume $n \geq 12$.

*If $\mu(n/4)=0$  we have $\mu(n/d)=\varphi(n/d)=1$. So $n\mid k+1$, which is impossible (as before).

*If $\mu(n/d) \neq 0 \neq \mu(n/4)$ then from $4 \mid n$, we have $2\mid d$. We have that $n/d$ and $n/4$ are square-free, and $\varphi(n/d) \mid \varphi(n/2) = \varphi(n/4)$. For the $2$-adic valuation of the LHS to be $0$, the denominators in the LHS must have the same number of factors $2$. This can only happen if $d \mid 4$. I.e. $d=2$ or $d=4$. The equation becomes
$$ \mu(n/d) \pm  \frac n 4 \mu(n/4) = \varphi(n/4) \, . $$
If $n/4$ has more than one prime divisor, $\frac n4 \geq \varphi(n/4) + 2$ and there are no solutions. Let $n=4p$, $p \geq 3$ prime. We have
$$ \mu(4p/d) \mp \, p = p-1 \, . $$
The sign must be $+$, so that $n\equiv12\pmod{16}$, hence $p \equiv 3 \pmod 4$. We also see that $\mu(4p/d) = -1$, so that $d = 4$.



Step 5. $n = 4$ or $n = 12$.
Proof. We have
$$ \zeta_{4p}^{k+1} + \zeta_{4p}^{-k-1} - p \zeta_{4p}^{p+1} + p \zeta_{4p}^{p-1} = 2 \, .$$
Write $k+1 = 4\ell$ and use that $\zeta_{2p} = - \zeta_p^{(p+1)/2}$:
$$ \zeta_p^\ell + \zeta_p^{- \ell} - p \zeta_p^{(p+1)/4} - p \zeta_p^{(p^2-1)/4} - 2 = 0 \, . $$
Because $[\mathbb Q(\zeta_p) : \mathbb Q] = p-1$, the $\zeta_p^{j}$ for $1 \leq j < p-1$ are linearly independent over $\mathbb Q$. That is, the only linear relation between the $(\zeta_p^j)_{0 \leq j \leq p-1}$ is that their sum is $0$. When $p = 3$ and $\ell = 1$, the above equality is exactly that linear relation. If $p > 5$, the coefficients of all $\zeta_p^j$ in the above equality must be $0$. But because $\ell \neq 0$, there is only one term involving $\zeta_p^0$, so there are no solutions. When $p = 5$, it must be that all five terms are of the form $c \zeta_p^{j}$ for a fixed $c \in \mathbb Q$, but that is not the case.
