Why relative velocity must be perpendicular to velocity of one of the bodies for closest approach I came across this question that asked the same, but I couldn't understand the reasoning. I feel it wasn't answered clearly, and the OP also sought clarification.
The answer stated,

From a physics perspective -- (here we go again, answering physics questions on a math site! ;) ) -- it may make more sense if you go into the inertial reference frame where the ship being approached is stationary. This involves subtracting the (vector) velocity of the ship being approached from the velocities of both ships. [...] one ship is stationary, its velocity is zero, so it stays at the same place all the time. It's at a point. (The red point.) The other ship moves along some straight line path past that point. The minimum distance from the point to the line is a perpendicular.

But this doesn't explain why $V_{ba} \perp V_{b}$. In fact, I think OP tried to ask the same.

But the question I'm asking John is marked in red above. The two lines in your diagram are perpendicular, the solid blue line represents the velocity vector of the patrol boat B as you rightly said. But this must mean that the dashed blue line is a relative velocity, (as well as being the shortest distance). Can you explain if the dashed line represents the velocity of B relative to S?

This is what I understand so far: (1-3 are covered in the linked question's accepted answer)

*

*Assume $V_{a}$ is velocity of first body and $V_{b}$ the velocity of the second

*$V_{ba} = V_{b} - V_{a}$ can be taken as object A is stationary and object B moves with $V_{ba}$

*For distance of closest approach, it's as simple as dropping a perpendicular from A to $V_{ba}$

*This part, I don't get (and unexplained in the linked question). For closest approach to happen, $V_{ba} \cdot V_{b} = 0$.

Why must $V_{ba} \perp V_{b}$?
Given $V_{ba}$ and $V_{b}$ are in different reference frames, how do I derive the relation that their dot product must be zero?
 A: In my opinion, you are right to question the answer key shown in Why must closest approach occur when relative velocity is perpendicular to motion?
It gives an incomplete explanation of how the patrol boat selects the
course to minimize the closest approach to the ship.
Provided that the patrol boat's course brings it closer to the ship than it was at the start, we can show that at the instant when the distance is minimized,
the relative velocity of the boat and ship is perpendicular to the line joining their two positions at that time.
We have seen this proved in multiple ways.
But the original question is completely unconcerned with what happens at the instant of closest approach.
To answer the actual question, which is to find the  direction the patrol boat should go in order to minimize the distance at closest approach,
notice that the difference of the two velocity vectors,
$V_B - V_S,$ is the velocity of the boat relative to the ship.
The direction of $V_S$ is fixed, but the direction  of $V_B$ can be chosen to be any direction.
Once you choose the direction of $V_B,$ the direction of $V_B - V_S$
is determined.
In the first part of the question, where the boat's speed is greater than the ship's, the tip of $V_B$ can fall anywhere on the circle shown in the figure below.
Since the tip of $V_S$ is inside that circle, the difference vector
$V_B - V_S$ (shown in red) can point in any direction.

To intercept a ship at any initial relative position $S$, you need the relative velocity $V_B - V_S$ to point in exactly the direction from $B$ to $S$, as shown for one choice of $V_B$ in the figure.
The desired direction of $V_B$ is determined by the point where the relative position vector from $B$ to $S$ intersects the circle.
If the boat is slower than the ship, we have a different set of possibilities,
as illustrated in the figure below.
Again, the tip of $V_B$ can be anywhere on the circle, and so can the tip of the relative velocity vector $V_B - V_S,$
but the direction of $V_B - V_S$ from $B$ is now limited to the directions between the two rays tangent to the circle.

We now have three possible cases, depending on the initial position of the ship $S$ relative to the boat $B.$
Case $1$.
If the direction from $B$ to $S$ is between those two tangents, as is the case if the ship is initially at $S_1,$ then it is still possible for the boat to intercept the ship. In fact there are usually two possible choices for the boat's direction in this case, though only one is shown here.
(This answer
shows both choices in a slightly different visualization.)
Case $2$.
If the ship is initially at $S_2,$ or anywhere else in the region between the rays from $B$ perpendicular to the tangents, the closest point of approach is at the starting condition; no matter which way the boat is headed, the relative velocity
$V_B - V_S$ will immediately take the boat farther from the ship and will continue to increase the distance between them indefinitely.
Case $3$.
If the ship is initially at a point such as $S_3,$ on the opposite side of one of the tangents as the circle but at a direction from $B$ not more than $90$ degrees away from the tangent direction, the closest the boat can get to the ship is at the relative position indicated by the dotted line to $S_3$ in the figure,
which connects $S_3$ to the nearest point on the nearby tangent to the circle.
The dotted line is perpendicular to the relative velocity vector along that tangent line, of course; but the fact that is relevant to the solution is that the tangent line is perpendicular to the radius that meets it, and that radius is the velocity vector $V_B$ that minimizes the distance at closest approach.
Therefore, in Case $3$, we want to choose a direction for $V_B$ that will be perpendicular to $V_B - V_S,$ and therefore we set up the right triangle shown in the second part of the answer key in Why must closest approach occur when relative velocity is perpendicular to motion?
What the answer key does not explain is how we know that Case $3$ applies.
One way to determine which case applies is to construct the diagram as if
Case $3$ applied, and then compare the direction of the relative velocity
$V_B - V_S$ in that diagram to the initial direction from $B$ to $S$.
If the direction from $B$ to $S$ goes through the triangle, Case $1$ applies and the boat can actually intercept the ship; if the direction is more than $90$ degrees on the other side of the relative velocity vector, Case $2$ applies and there is no way to get closer to the ship no matter which way the boat heads;
otherwise Case $3$ applies.
Examining the angles found in the answer key, the direction of $V_B - V_S$ is
$108.6$ degrees, which is less than $90$ degrees away from the initial direction from the boat to the ship ($50$ degrees), so Case $3$ applies.
A: We have two boats $A$ and $B$ in the plane  with
$$A(t)=a+t u,\qquad B(t)=b+t v\qquad(-\infty<t<\infty)\ ,$$
whereby it is assumed that the velocities $u$ and $v$ are linearly independent. We are interested in the vector
$$d(t):=B(t)-A(t)=(b-a)+t(v-u)\ .$$
Since $\bigl|d(t)\bigr|\to\infty$ when $|t|\to\infty$ there is a certain $t_*$ for which $|d(t)|$ is minimal, and this $t_*$ can be found by putting
$${d\over dt}|d(t)|^2={d\over dt}\bigl(d(t)\cdot d(t)\bigr)=2 d(t)\cdot d'(t)$$
to zero. As $d'(t)=v-u$ it follows that
$$d(t_*)\cdot (v-u)=0\ .$$
This means that at the time $t_*$ of "nearest contact" the relative speed of the two boats is orthogonal to the vector $d(t_*)$ connecting the two boats.
