Is it possible to formalize the n queen problem as a continuous function optimization problem? The goal is to find one possible placement with n queen problem. But is there a smart way to reformat the problem so that we can use function optimization technique?
 A: As @vonbrand mentioned, you can model this with a binary variable for each cell.  Let $x_{i,j}$ indicate whether cell $(i,j)$ contains a queen.  The constraints are linear:
\begin{align}
\sum_j x_{i,j} &= 1 &&\text{for all $i$}\\
\sum_i x_{i,j} &= 1 &&\text{for all $j$}\\
\sum_i x_{i,i} &\le 1\\
\sum_i x_{i,n-i+1} &\le 1
\end{align}
If you want to avoid explicit integrality restrictions, you can impose $x_{i,j}(1-x_{i,j})=0$, but these nonlinear constraints are nonconvex.  Better to omit them and use an integer linear programming solver.
A: The classic solution (e.g. as given by Dijkstra and Wirth, and commonly used as an example of backtracking) is a recursive program. It can be made more efficient (try first the most resticted next column; or the row in the next column that restricts the following most, or covers more of the remaining free cells, are obvious heuristics if you just look for a solution, not all of them; but the gain might be less than the simple algorithm).
Representing the restictions (which by their very nature are discrete) doesn't look easy. You might try to model them as a (nonlinear) integer program, i.e. restriction of 0 or 1 queen on each diagonal, one on each row and column, and try solving a continuous analogue. Looks messy, and there is no reason to believe a continuous solution can be rounded to a solution to the discrete problem.
