Nearest point to point A on polygon P in 3D We have given a polygon $P$ which consists of $4$ 3D points $p_1$, $p_2$, $p_3$, $p_4$, the polygon is not necessarily flat. Then let us have a point $A$ (in my case this point is something like a center of mass to polygon $P$ , but the point is not always placed on the polygon). I need to find point $X \in P$ which is the nearest point to point $A$ . 
Any suggestions? 
 A: This problem can be formulated as a quadratic programming task. Let us introduce a point $\boldsymbol{a}$ and a general point $\boldsymbol{x}$. Then the objective function is given by
$$F(\boldsymbol{x}|\boldsymbol{a})=[\boldsymbol{x}-\boldsymbol{a}]^T[\boldsymbol{x}-\boldsymbol{a}].\qquad (*)$$
We additionally need to assume that the polygon is flat. If it is not flat we will need to further differentiate the problem. Then we determine the normal vector $\boldsymbol{n}$ of the polygon by using the cross product of two vectors which are aligned with the edges of the polygon. Then we know that every point in the polygon needs to fulfill the following equality constraint
$$\boldsymbol{n}^T[\boldsymbol{x}-\boldsymbol{p}_1 ]=0,\qquad (**)$$
in which $\boldsymbol{p}_1$ is a corner point of the polygon. Then we construct the outward edge normal for all the edges I will call them $\boldsymbol{n}_1$ for the edge $\boldsymbol{p}_1$ to $\boldsymbol{p}_2$, $\boldsymbol{n}_2$ for the edge $\boldsymbol{p}_2$ to $\boldsymbol{p}_3$, and so forth. In order to achieve this we will need to take the cross product of the polygon normal $\boldsymbol{n}$ and the edge vectors.
Then we use these outward normal vectors to span an additional hyperplane for each edge. We will have the following inequality
$$\boldsymbol{n}^T_i[\boldsymbol{x}-\boldsymbol{p}_i]\geq 0.\qquad (***).$$
Then we can formulate our quadratic programming problem as
$$\text{minimize } [\boldsymbol{x}-\boldsymbol{a}]^T[\boldsymbol{x}-\boldsymbol{a}]$$
$$\text{subject to: } \boldsymbol{n}^T[\boldsymbol{x}-\boldsymbol{p}_1 ]=0 \quad \text{and} \quad
\boldsymbol{n}^T_i[\boldsymbol{x}-\boldsymbol{p}_i]\geq 0 \quad \forall i=1,2,3,4$$ 
