In $\mathbb R^3$ given a plane and a point belong to it, in this plane I have a rectangle determined by four vertices and want to check if the given point locates inside the rectangle

In 2D it's known that I can check that the point is located inside the rectangle by check the result of the dot product with lower line and the point, and upper line and the point, if the signs of to tow results is opposite then the point is inside the rectangle (and of course check the same in right and left lines of the rectangle), is the same thing correct in the 3D plane ?

  • $\begingroup$ Since you know the point is in the plane you can project the coordinates of the point and the rectangle onto $\mathbb R^2$ and use your planar techniques. $\endgroup$
    – John Douma
    Jun 28, 2021 at 20:24
  • $\begingroup$ I thought in this solution, but don't know how to project them onto 2D @JohnDouma $\endgroup$ Jun 28, 2021 at 20:41
  • $\begingroup$ As long as the plane is not parallel to the $z$ axis, you can project by simply dropping all the z coordinates. An inside point will be inside in the x-y plane projection. An outside point will be outside. This will, however, turn your rectangle into a more general parallelogram, if you can handle that. If the plane is paralle to $z$, then drop, say, all the x coordinates. $\endgroup$ Jun 28, 2021 at 21:05

2 Answers 2


I did not understand how your method works in 2D with just two dot products. It seems to me that you need to check the other sides as well.

But the idea works in 3D as well. If you have the rectangle $ABCD$ and a point $P$ in that plane, then the angle between $AB$ and $BP$ has to be between $0$ and $90^\circ$. That's equivalent of saying $$\vec{AB}\cdot\vec{BP}<0$$ You also need to apply the same for all sides: $$\vec{BC}\cdot\vec{CP}<0\\\vec{CD}\cdot\vec{DP}<0\\\vec{DA}\cdot\vec{AP}<0$$

  • $\begingroup$ Thank you, another question, is the idea still work with any quadrilateral ? still work with triangle ? $\endgroup$ Jun 28, 2021 at 21:50
  • $\begingroup$ Not exactly. I relied on the fact that the angles are $90^\circ$, and that's where the dot product changes sign. It will not even work for a parallelogram that is not rectangle. $\endgroup$
    – Andrei
    Jun 28, 2021 at 21:52
  • $\begingroup$ Is there any standard way to know if the point belongs to any geometrical shape ? $\endgroup$ Jun 28, 2021 at 21:56
  • $\begingroup$ There are several methods, as seen in en.wikipedia.org/wiki/Point_in_polygon $\endgroup$
    – Andrei
    Jun 29, 2021 at 2:41

In the plane. when is a point $(a,b)$ inside a convex polygon?
Solution: For every consecutive triple of boundary vertices $U, V, W,\,$ solve $\ s\vec{VU}+t\vec{VW}=(a,b)-V.\ $ If, in every case, both $s$ and $t$ are nonnegative, then $(a,b)$ is inside the polygon. (Any vertices with an interior angle of 180 degrees are ignored.)

  • $\begingroup$ Thank you, but I solved it, the algorithm that I used is as following : 1- draw a line from the point to every vertex of the polygon then some triangles will be formed 2- calc the sum of the areas of all these triangles 3- repeat from 1 but with some point from the polygon(I choosed a vertex from it). compare to sums of areas if equal then it's inside otherwise it isn't inside is that true ? $\endgroup$ Jun 29, 2021 at 0:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .