Law of cosines with impossible triangles Is there any mathematical significance to the fact that the law of cosines...
$$
\cos(\textrm{angle between }a\textrm{ and }b) = \frac{a^2 + b^2 - c^2}{2ab}
$$
... for an impossible triangle yields a cosine $< -1$ (when $c > a+b$), or $> 1$ (when $c < \left|a-b\right|$)
For example, $a = 3$, $b = 4$, $c = 8$ yields $\cos(\textrm{angle }ab) = -39/24$.
Or $a = 3$, $b = 5$, $c = 1$ yields $\cos(\textrm{angle }ab) = 33/30$.
Something to do with hyperbolic geometry/cosines?
 A: This is not a directly a matter of hyperbolic geometry but of complex Euclidean geometry.  The construction of "impossible" triangles is the same as the construction of square roots of negative numbers, when considering the coordinates the vertices of those triangles must have.  If you calculate the coordinates of a triangle with sides 1,3,5 or 3,4,8 you get complex numbers.  In ordinary real-coordinate Euclidean geometry this means there is no such triangle.  If complex coordinates are permitted, the triangle exists, but not all its points are visible in drawings that only represent the real points.
In plane analytic geometry where the Cartesian coordinates are allowed to be complex, concepts of point,line, circle, squared distance, dot-product, and (with suitable definitions) angle and cosine can be interpreted using the same formulas.  This semantics extends the visible (real-coordinate) Euclidean geometry to one where any two circles intersect, but possibly at points with complex coordinates.  We "see" only the subset of points with real coordinates, but the construction that builds a triangle with given distances between the sides continues to work smoothly, and some formulations of the law of Cosines will continue to hold.
There are certainly relations of this picture to hyperbolic geometry.  One is that $cos(z)=cosh(iz)$ so you can see the hyperbolic cosine and cosine as the same once complex coordinates are permitted.  Another is that the Pythagorean metric on the complex plane, considered as a 4-dimensional real space, is of the form $x^2 + y^2 - w^2 - u^2$, so that the locus of complex points at distance $0$ from the origin contains copies of the hyperboloid model of hyperbolic geometry.  But there is no embedding of the hyperbolic plane as a linear subspace of the complex Euclidean plane, so we don't get from this an easier way of thinking about hyperbolic geometry.
To help visualize what is going on it is illuminating to calculate the coordinates of a triangle with sides 3,4,8 or other impossible case, and the dot-products of the vectors involved.
A: One can prove the triangle inequality in any abstract inner product space, such as a Hilbert space; it is a consequence of the Cauchy-Schwarz inequality $\langle a, b \rangle^2 \le ||a||^2 ||b||^2$.  In order for the triangle inequality to fail, the Cauchy-Schwarz inequality has to fail, and in order for Cauchy-Schwarz to fail (which corresponds to the "cosine" no longer being between $1$ and $-1$), some axiom of an abstract inner product space has to be given up.  One choice is to give up positive-definiteness; in other words, lengths of vectors are no longer always non-negative.  This leads to geometries like Minkowski spacetime which are relevant to relativity.  In Minkowski spacetime, there is a reverse triangle inequality instead.
Edit:  I should also mention that the "unit sphere" in Minkowski spacetime is a model for hyperbolic geometry called the hyperboloid model.
A: For some a, b, and c that form a triangle: increasing the length of c increases the measure of angle C and as m∠C approaches 180°, cos C approaches -1; decreasing the length of c increases the measure of angle C and as m∠C approaches 0°, cos C approaches 1.  Extending this pattern, it makes sense that if c > a + b, c has gotten bigger past making a triangle with a and b, so cos C < -1, and if c < |a-b|, c has gotten smaller past making a triangle with a and b, so cos C > 1.
In hyperbolic geometry, the definition of lines (and hence triangles) is different and the sum of the measures of the angles in a triangle is less than 180°.  There is a Hyperbolic Law of Cosines, but it's not quite the same.
I don't think there's a sensible way to relate hyperbolic cosine to the Law of Cosines in Euclidean geometry.
