Best way to do this is to use the signed magnitude of cross product of vectors, i.e. $$(x_1,y_1)\times(x_2,y_2) = x_1 y_2 - x_2 y_1 .$$
This formula equals to $0$ if and only if $(x_1, y_1)$, $(x_2,y_2)$ and $(0,0)$ are on the same line. Points $p$, $q$, $r$ are on the same line if and only if $p-r$, $q-r$ and $r-r$ are on the same line.
To check if point is inside a rectangle you can use the cross product too: its sign depends on whether point is on the left or on the right of the line. The point $(x_2, y_2)$ is on the left of line from $(0,0)$ to $(x_1,y_1)$ if and only if $$ x_1 y_2 - x_2 y_1 > 0.$$
So to check if the point is in the rectangle just check if it's on the same side of every segment, i.e. if $a,b,c,d$ are consecutive vertices of the shape and $x$ is the point in question, then
will have the same sign if and only if $x$ is inside the rectangle. If $a, b, c, d$ are sorted counterclockwise (clockwise) then all have to be positive (negative).
Hope that helps ;-)