So I'm reading about the history of hyperbolic geometry and something like this came up: "two thousand years later, people gave up on trying to derive the fifth postulate from the other 4 and begun studying the consequences of the mathematical structure without the fifth postulate. The result is a coherent theory". My question: were there any possibility of incoherence? I understand some postulates are actually definition of objects, then, clearly, other postulates might be dependent on this other postulate, but it doesn't seem to be the case.
When people say that noneuclidean geometry is equiconsistent with Euclidean geometry, and therefore all those old Renaissance proofs of the parallel postulate were incorrect, actually they're oversimplifying quite a bit, and taking things out of historical context.
Here is a sketch of a proof that elliptic geometry is inconsistent. In elliptic geometry, we have an axiom that says that for a given line L and a point p not on the line, no line through p exists that is parallel to L. From Euclid's first four postulates plus this non-parallelism postulate, we can prove that there is an upper limit on the area of any figure. But then that contradicts the third postulate, which says that we can construct a circle with any given center and radius, since according to the second postulate the radius can be made as big as desired.
Is this proof fallacious? Well, that depends completely on your interpretations of the second and third postulates. For some perfectly reasonable interpretations of them, it's a valid proof. Euclid wrote the postulates in language that was a model of rigor for its time, but before the consistency of noneuclidean geometry could really be addressed, people needed to decide on a disambiguation of them.
So, my answer to the question "was there any possibility of incoherence?" is that yes, there certainly was. It depended completely on the interpretatation of the postulates, which was not at all fixed to a sufficient level of rigor.
There is also the question of whether the first four postulates were consistent, and although nobody has ever believed that they weren't, it was only relatively recently in history that anything was really done to address this question. Tarski proved that in first-order logic, Euclidean geometry was consistent. (This is not a violation of Godel's theorems, because Godel's theorems only apply to theories that can describe a certain amount of arithmetic, which first-order Euclidean geometry can't do.) Furthermore, we now know that if Euclidean geometry is inconsistent, the real number system must also be inconsistent, which seems pretty unlikely; but this is an idea that would not have occurred to anyone before the invention of Cartesian geometry and the 19th-century formalization of the real number system.
There are a few different issues involved. I do not know as much about Lobachevsky's attitude, but Bolyai and Gauss were a bit timid about claiming complete truth for what they had discovered. Gauss was so afraid of criticism that he did not publish his findings. Bolyai believed he had a genuine construction. In fact, everybody else doubted the validity of the hyperbolic plane until the models were published. We now know that Beltrami derived pretty much everything first. However, he did not emphasize the simpler version in only 2 dimensions, so history calls those the Poincare disk and the Poincare upper half plane. I do not know whether Beltrami noticed the relationship with complex variables in dimension 2, so that may be the more important reason for attaching the name Poincare.
As to axioms and dependence, it is generally the case that no proof of validity for a system can be found within the system itself. So, the statement these days is that the geometry of the hyperbolic plane is valid if and only if Euclidean geometry is valid. Nobody actually doubts the consistency of Euclidean geometry, but there is the matter of what we can prove. In the same light, my impression is that we cannot prove the consistency of set theory. That's life.
Anyway, I imagine you have Stillwell's book with original sources. For convenience, see the pdf link in GREENBERG See also Geometry: Euclid and Beyond by Robin Hartshorne and Euclidean and Non-Euclidean Geometries by Marvin Jay Greenberg. My impression is that you have a way to go, but the highlight of this is the axiomatic treatment of Hilbert's Field of Ends, not treated at an introductory level in too many places. In short, Bolyai, Gauss, and Lobachevsky (to a lesser degree) worked in the synthetic style, and people started to believe in it only after the models over the real numbers were discovered. Hilbert asked for an entirely synthetic approach (as for the ancient Greeks), no real numbers. This was done.
There was hope that such a theory would eventually go on to derive the 5th postulate and show its not really necessary for it to be stated independently, but that it is dependent on the first four postulates. However, there were 3 possible scenarios derived from only (leading to spherical geometry, euclidean geometry and hyperbolic geometry if I remember correctly) the first 4 postulates, only one of which was the 5th postulate.
There was possibility, which we now know doesn't exist, that by only taking the first 4 postulates, some very wild and absurd consequences would arise, and many mathematicians of that time really believed that some consequences were so absurd as to be proof by contradiction that 5th postulate is really needed for coherent theory (which form today's standpoint only means that it was in accordance with intuition).