Working in two dimensions I have a point $P$, a line $AB$ defined by two other points $A$ and $B$, with $P$ not on $AB$. From another set of points, I need to find the nearest point to $P$ which is on the same side of the line $AB$ as $P$.
Working out the distances to $P$ is easy, and there are a number of formulae which give the distances to the line $AB$; but they don't give a sign which lets me know which side of $AB$ a point is on. I'm guessing that I can get this answer out by neglecting to take the square root in one of the formulae, and so I can just order the points by the squared distance and look only at the ones where it comes out with the same sign as the answer for $P$.
If this is the case, what's a suitable formula for the squared distance between a point $(x,y)$ and the line between $A=(x_a, y_a)$ and $B=(x_b, b_y)$ in terms of $x, y, x_a, y_a, x_b$ and $y_b$?