If you actually want to check the distance to the segment and not to the line, you will have to check that it is not "beyond" one of the endpoints. That is, that the orthogonal projection of the point $x_0$ is between $x_1$ and $x_2$. If not, the answer will be the distance to the closest of the two endpoints.
Here is some pseudocode:
input vectors x0,x1,x2
// if x0,x1,x2 form an obtuse angle, then x0 is closest to x1
if dotprod(x0-x1,x2-x1) < 0 then
// if x0,x2,x1 form an obtuse angle, then x0 is closest to x2
else if dotprod(x0-x2,x1-x2) < 0 then
Cross products and dot products should be relatively inexpensive in terms of compute cost, as they are only a handful of floating point operations each. I know nothing about matlab, but you might want to check if another language would yield performance increases.