Let $(x_1,y_1),(x_2,y_2),(x_3,y_3),(x_4,y_4)$ be the vertices of a quadrilateral. How could I know that an arbitrary point $(x_5,y_5)$ is inside this quadrilateral or outside? Is there any formula to do this?
One way is to calculate area of quadrilateral $\square P_1P_2P_3P_4$ and four triangles $\triangle P_1P_2P_5, \triangle P_2P_3P_5, \triangle P_3P_4P_5, \triangle P_4P_1P_5$ by Shoelace formula. If sum of areas of four triangles $ = $ area of quadrilateral, then the point $P_5$ is inside.
I think this works only for convex quadrilaterals. Works for convex as well as concave quadrilaterals.
Triangulate the quadrilateral, i.e., draw a diagonal. First check whether the point lies within the first triangle, if not then check whether it lies in the second triangle. Note that if the equation of a straight line is $ax+by+c=0$, and suppose that it divides the plane into two half planes, $H_1$ and $H_2$, then for two points $(x_1,y_1)\in H_1$, $(x_2,y_2)\in H_2$, $(ax_1+by_1+c)(ax_2+by_2+c)<0$. Using this fact can you now find out how to check whether a point lies inside a triangle whose vertices are given?