No of triangles in a square which contains all the m points? Given a square $A(0,0)$, $B(0,n)$, $C(n,n)$ and $D(0,n)$ in $X­Y$ plane and a set of $m$ points. The $m$ points strictly lie inside the square $ABCD$. It is clear that there are $4n$ integer points on the contour of the square $ABCD$. Count the number of triangles whose vertices are chosen from the $4n$ points and enclose all the $m$ points.
 A: Write the 3 lines defining the sides of the triangle in terms of the vertices, and write each line as $ax + by = c$ where $ax + by \leq c$ defines the interior of the triangle. Then you want the maximum of $ax + by$ over the $m$ points to be less than or equal to $c$ for each side of the triangle. This is the if and only if condition for the $m$ points to all be inside the triangle. This should allow you to characterize and count the triangles containing the points potentially without having to try them all one by one.
A: Here’s another tip that might help.
Given a triangle $ABC$, suppose you want to know if the point $P$ is inside or outside the triangle. Compute this sum of three directed angle measurements: $\angle APB + \angle BPA + \angle CPA$. If the point is inside the triangle, this sum will be $\pm2\pi$. If the point is outside the triangle, the sum will be $0$. (Up to the $2\pi$ multiple, this is the “winding number” of the oriented triangle about the point.)
You must be careful to compute the directed angles, which can be between $-\pi$ and $\pi$. One way to do it is with the two-argument inverse tangent function called $\mbox{atan2}$ in many programming languages. The directed angle (the bearing) from $\vec u = (x_1,y_1)$ to $\vec v=(x_2,y_2)$ is $\mbox{atan2}(\vec{u}\cdot\vec{v},|\vec{u}\times\vec{v}|)$, where $\vec{u}\cdot\vec{v}=x_1x_2+y_1y_2$ and $|\vec{u}\times\vec{v}|=x_1y_2-y_1x_2$ are the dot product and the magnitude of the cross product. 
