This is entirely determined by what we consider a rotation in the plane.
Let $u, v$ be two unit vectors. Let the angle between them be $\theta$, and we can naturally write $v$ as
$$v = u \cos \theta + u_\perp \sin \theta$$
where $u_\perp$ is a unit vector perpendicular to $u$. Then clearly, the dot product is $u \cdot v = \cos \theta$.
But, if you're not in a Euclidean plane anymore, this relationship no longer holds. For example, in a Lorentzian space, instead of cosine and sine, we get hyperbolic functions instead:
$$v = u \cosh \theta + u_\perp \sinh \theta$$
And the dot product is $u \cdot v = \cosh \theta$. A physicist should recognize that this $\theta$ is the "rapidity", and that the form of $v$ given here is exactly that of any Lorentz boost.
So the reason we use sine and cosine in Euclidean space is because they are dictated by the use of sine and cosine in rotations. In other spaces, with different rotation operators, you use the functions that are associated with those rotations instead.