Why did Fermat care about characterizing primes on the form $p=x^{2}+ny^{2}$?

Question

Im currently trying to figure out the genesis of quadratic reprocity by using Cox and Lemmermeyers books. I also got a copy of some works of Fermat but it is in German. It seems like there is some connection with basic geometry but I can really figure it out.

I made one oberservation and that was that if $p=x^{2}+y^{2}$ and since $$(a^{2}+b^{2})(c^{2}+d^{2})=(ac \pm bd)^{2}+(ad \pm bc)^{2}$$ then we can decompose the hypotenuse of the Pythagorean theorem using fundamental theoreom of arithmetic. This only applies to right-angled triangles and Fermat seams to deal with ellipses. My own answer if it is even valid is therefore not satisfactory.

Answer 1

The question in the title is very interesting, but you may want to consider focusing your text a little. You seem to be asking three questions at the same time, not clearly delineated.

1. The question in the title: why did Fermat care about representing primes by these quadratic forms?
2. What are the origins of quadratic reciprocity?
3. I do not understand the third question, but I am guessing something like a factorization for $x^2+ny^2$ similar to the one you provide in your post (you are looking for Brahmagupta's identity).

The first two questions, and most likely the third one as well, are answered in A. Weil's book Number Theory: an Approach Through History. More specifically, Sections V through VIII of Fermat's chapter describe Fermat's (and others') attempts to extend Fermat's result on sums of squares to more general forms $x^2+ny^2$ in an attempt to find large prime numbers and more conceptually to understand some confusing passages in Diophantus' Arithmetica (which forms the inspiration for most of Fermat's work in the first place). Some transformations Diophantus takes for granted turn out to hinge on properties of quadratic forms of the form $x^2+2y^2$ and $x^2+43y^2$.

Fermat's observation that the expressibility of a prime as a sum of squares only depends on congruence properties of the prime led him to ask the corresponding question for other forms (remember: during his study of Arithmetica and in connection to finding large primes): does the candidate expression $p=x^2+2y^2$ depend only on a congruence $p$ has to satisfy? More generally, does $p=x^2+ny^2$ depend only on a congruence related to $n$? This question, although harder than the reciprocity law, formed one of the (many!) seeds for the formulation of quadratic reciprocity. But it was not Fermat who made these connections (Fermat did not study the general form $x^2+ny^2$), it was Euler. In Euler's chapter of Weil's book, sections VIII and further, describe Euler's meticulous analysis of these quadratic forms, his encounter with what we call Dirichlet characters today and after a very long time, a formulation of the quadratic reciprocity law (along with Legendre's). In particular, you should read very carefully Legendre's chapter, because this is where all the threads concerning quadratic reciprocity come together in Weil's book.

Weil's book is well written, contains a wealth of historical data from which I merely (and perhaps somewhat inaccurately) sampled for the answer, and I suggest you read it carefully to answer your questions in full detail.