Why do we need prime ideals in the spectrum of a ring? I'm reading Atiyah Macdonald, where they introduce in the exercises of chapter one a topological space $\operatorname{Spec}(A)$ associated to a ring $A$, which is defined as
$\operatorname{Spec}(A) \equiv \{ I : \text{I is a prime ideal in A} \}$.
I have some questions about this topological space:

*

*Why should the ideals be prime? As far as I can tell, it seems to be a technical condition to allow union of open sets to work. Is there a deeper reason?

*Why do we generate the closed sets as collections of prime ideals? As far as I can tell, there is nothing that breaks with infinite union and intersection, so we can just as well take the sets to be open?

*Why is the function that takes subsets of the ring to a closed set called $V$ in the text? It cannot be 'variety': it is taking elements/points (geometry) into ideals (algebra). If anything, it is an "anti-variety".

Perhaps I have missed something in proving that $\operatorname{Spec}(A)$ is a topological space, so I will recapitulate proof sketches below. First, the topology on $\operatorname{Spec}(A)$ is given by stating that the closed sets of the topology are given by:
$$
V: 2^A \rightarrow 2^{\operatorname{Spec}}; 
V(S) \equiv \{ I \in \operatorname{Spec}(A): S \subseteq I\} \\ 
 \tau_\text{closed} \equiv \{ V(S): S \subseteq A \}
$$
That is, for every subset $S$ of $A$, the set of prime ideals that contain $S$ [which is denoted as $V(S)$] is a closed set. Now we check that:

*

*$\emptyset \in \tau$ since $V(R) = \{ I \in \operatorname{Spec}(A) : R \subseteq I \} = \emptyset$ [no proper ideal contains the whole ring]

*$\operatorname{Spec}(A) \in \tau$ since $V(\{0\})= \{ I \in \operatorname{Spec}(A) : \{ 0 \} \subseteq I \} = \operatorname{Spec}(A)$ [every ideal contains zero]

*Intersection:

$$
V(S) \cap V(S') = \{I \in \operatorname{Spec}(A): S \subseteq I\} \cap \{I \in \operatorname{Spec}(A): S' \subseteq I\} \\
= \{ I \in \operatorname{Spec}(A): S \cup S' \subseteq I \} = V(S \cup S')
$$
4. Union [The part where prime matters]:
$$
\begin{align*}
&V(S) \cup V(S') = \{I \in \operatorname{Spec}(A): S \subseteq I\} \cup \{I \in \operatorname{Spec}(A): S' \subseteq I \} \\
&= \{I \in \operatorname{Spec}(A): S \subseteq I \lor S' \subseteq I\} \\
&= \{I \in \operatorname{Spec}(A): S S' \subseteq I\} \ \ \text{(Since $I$ is prime, $ss' \in I \implies s \in I \lor s' \in I)$} \\
&= V(SS')
\end{align*}
$$
This union of $V(\cdot)$s should also work with infinite unions, since we will get $\cup_i V(S_i) = \prod_i  S_i$. I suppose the problem is that we do not have a topology on $A$ to define infinite products of elements? If so, does this construction work in a ring that possesses a topology to talk about infinite products?
 A: The historical genesis of algebraic geometry is considering the solutions of some finite collection of polynomials inside $k^n$ for $k$ an algebraically closed field. One can check that in this scenario that the irreducible closed sets which are given by the vanishing locus of a finite collection of polynomials exactly correspond to the prime ideals of $k[x_1,\cdots,x_n]$. So if we want to try and generalize beyond $k^n$, this would be a good avenue to explore. More details on wikipedia and probably in every algebraic geometry book.
To see why taking the closed subsets of $\operatorname{Spec} A$ to be $V(I):=\{\mathfrak{p}\in A\mid I\subset \mathfrak{p}\}$ behaves appropriately under arbitrary intersection, let $\{I_t\}_{t\in T}$ be a family of ideals of $A$. Then $\bigcap_{t\in T} V(I_t)$ is the collection of prime ideals which contain all of these $I_t$, which is equivalent to the prime ideals containing the sum $\sum_{t\in T} I_t$. Since the sum of ideals is always an ideal, we see that the sets of the form $V(I)$ are closed under arbitrary intersection and $\bigcap_{t\in T} V(I_t)=V(\sum_{t\in T} I_t)$. On the other hand, they do not behave correctly under arbitrary union: if $\{I_t\}_{t\in T}$ is as before but we take $\bigcup_{t\in T} V(I_t)$, we now want to think about the prime ideals which contain the intersection of all $I_t$. As mentioned in another answer, the ideals $I_t=(z-t)$ for $A=\Bbb C[z]$ and $t\in T=\Bbb Z$ have intersection zero, which is not equal to $\bigcup_{t\in T} V(I_t)$. So $\bigcup_{t\in T} V(I_t)\neq V(\bigcap_{t\in T} I_t)$ and it would be inappropriate to choose $V(I)$ to be the open subsets.
(I should also point out that your "proof" contains a couple rather serious mistake: the correct way to put ideals together is to take their sum, not their union. Also, the correct way to intersect ideals is to take their intersection, not their product. Both ideas work correctly in some cases but fail badly in general, and one should do the correct thing.)
For #3, the symbol "$V$" stands for "vanishing set". This started as a hold-over from the old days of algebraic geometry, because we would literally ask about where our collection of polynomials all vanished in $k^n$. In the modern language of schemes, in order to say that $\mathfrak{p}\in V(I)$, we can ask about when all elements of $I$ vanish in the ring $A_{\mathfrak{p}}/\mathfrak{p}$, the residue field at the point $\mathfrak{p}$ (where $\mathfrak{p}$ is a prime ideal). NB: once you start thinking about more than just sets, you'll need to be a little more careful here - see this recent answer of mine for more details if you wish.
A: It is not true that the union of an infinite family of sets of the form $V(I)$ is necessarily of the form $V(I)$. For example, let $R = \mathbb{C}[x]$ and let $I_n$ be the principal ideal generated by $(x-n)$. Then the intersection of all the $I_n$ as $n$ varies over the integers is zero (no nonzero polynomial has infinitely many roots), but the union of the $V(I_n)$ isn't all of $\text{Spec}(R)$ (since the polynomial $(x - 1/2)$ isn't in any of them for example).
A: To answer your first question why prime ideals are interesting.
The classic point of view is to studying the geometry over an algebraic closed field $k$ and focus on a subset $M \subset k[x_1, \ldots, x_n]$. We can view varieties as the set of points on which all functions vanish, that is $V(M) := \{(x_1, \ldots, x_n) \in k^n : f(x_1, \ldots, x_n) = 0 \text{ for all } f \in M\}$.
A corollary of one of the main results is that if $\mathfrak{a}$ is an ideal in $k[x_1, \ldots, x_n]$ then we have an equivalence $V(\mathfrak{a}) = \operatorname{Hom}_{k-alg}(A, k)$ where $A = k[x_1, \ldots, x_n]/\mathfrak{a}$.
Generalizing this into $\operatorname{Hom}_{Ring}(A, K)$ for general commutative rings $A$ and fields $K$ we can define $\operatorname{Spec} A$ to be the collection of equivalence classes of ring morphisms $A \rightarrow K$ for $K$ a field where two maps $A \rightarrow K$, $A \rightarrow K'$ are identified if there exists a ring morphism $K \rightarrow K'$ which makes the diagram commute.
$\hskip2in$
Technically this can be seen as a certain colimit.
This construction says nothing about prime ideals. But there is of course an identification between $\operatorname{Spec} A \longrightarrow \{\mathfrak{p} \subset A, \mathfrak{p} \text{ prime} \}$ given by the bijection of taking a map $(f\colon A \rightarrow K) \in \operatorname{Spec} A$ and sending it to its kernel $\operatorname{ker}(f) \subset A$.
Showing that there is an inverse uses the fact that $\mathfrak{p}$ being prime allows us to find a map $A \longrightarrow \operatorname{Frac}(A/\mathfrak{p})$ with the ring of fractions being a field.
A: The intuition for allowing only prime ideals is that with the spectrum we are trying to generalise the usual notion of an (affine) variety to arbitrary rings, not just polynomial rings over an algebraically closed field $k$.
Think about $k[t]$ over an algebraically closed field $k$ (the following works in much more generality but this is for illustration). There is a bijective correspondence between points of $\mathbb{A}^1$ and the maximal ideals of $k[t]$ namely via $x \mapsto (t - x)$. Thus it is natural when we extend this definition to not want to include much more than just the maximal ideals.
One might ask why do we not just think about the maximal spectrum then? Well for example over $\mathbb{Z}[t]$ we definitely want to keep all the interesting information about both prime numbers and polynomials. This is part of the beauty and power of $\operatorname{Spec}$, it allows us to put geometry and number theory on the same footing.
For your second question, this too is natural when we think about what we want the Zariski topology to look like - we want it to be correct in our example $\mathbb{A}^1$ (i.e., we want the topology we define on $\operatorname{Spec}(k[t])$ to look the the Zariski topology on $\mathbb{A}^1$).
We now see the answer to your third question! You were right, the reason we use $V$ is because it is a "variety" the closed sets in $\mathbb{A}^n$ are varieties!
Edit: I said a wrong thing.
A: Question: "Why should the ideals be prime? As far as I can tell, it seems to be a technical condition to allow union of open sets to work. Is there a deeper reason?"
Answer: If $k$ is any field and $A,B$ are finitely generated $k$-algebras and $f: A \rightarrow B$ a map of $k$-algebras, it follows for any maximal ideal $\mathfrak{m} \subseteq B$ the inverse image $f^*(\mathfrak{m}):=f^{-1}(\mathfrak{m}):=\mathfrak{n} \subseteq A$ is a maximal ideal. Hence if we let $Spec^m(A)$ denote the set of maximal ideals in $A$ with the "Zariski topology" we get for any map $f$ a continuous map of topological spaces
$$f^*: Spec^m(B) \rightarrow Spec^m(A)$$
with the property that for any finitely generated $K$-algebra $R$ and a map $g: R \rightarrow A$ it follows $(f\circ g)^*=f^* \circ g^*$. Hence we get a functor
$$F: \underline{k-alg} \rightarrow \underline{top}$$
from the category of finitely generated $k$-algebras and maps  to the category of topological spaces and continuous maps. In fact
$$F(A):=Spec^m(A) \subseteq Spec(A)$$
is a topological sub-space and $F$ is a "sub functor" of $Spec(A)$.
If $X^m:=Spec^m(A)$ it is in fact possible to construct a sheaf $\mathcal{O}_{X^m}$ on $X^m$ so that $(X^m, \mathcal{O}_{X^m})$ is a locally ringed space and a functor
$$F^m: \underline{k-alg} \rightarrow \underline{rspace},$$
where $F(A):=(X^m, \mathcal{O}_{X^m})$ and where $\underline{rspace}$ is the category of locally ringed spaces and maps (see link below). The sheaf $\mathcal{O}_{X^m}$ is defined as follows:
Definition: Let $i:X^m \rightarrow X$ be the canonical inclusion and define $\mathcal{O}_{X^m}:=i^{-1}(\mathcal{O}_X)$ as the topological inverse image.
The functor $F^m$ has the property that $F^m(A) \cong F^m(B)$ iff $A\cong B$ as $k$-algebras. Hence for finitely generated $k$-algebras (or more generally: schemes of finite type over a field or a Dedekind domain) you may construct a ringed topological space using only maximal ideals. A problem with this construction is that you need $A$ to be a finitely generated $k$-algebra for this to work: Such a ring is a "Hilbert-Jacobson ring", and what makes this work is that any prime ideal in such a ring is the intersection of maximal ideals.
Note: This construction does not assume that the ring $A$ is reduced or a  domain: The ring $A$ may be an arbitrary finitely generated $k$-algebra where $k$ is any Dedekind domain. Hence you may construct a ringed topological  space $(X^m, \mathcal{O}_{X^m})$ from a ring $A$ with nilpotent elements, and study "nilpotent phenomena" in "geometry". This was one of the original motivations of the introduction of the "scheme" - to systematically use nilpotent elements in geometry and number theory. Such "non-reduced" or "non-irreducible" schemes aries when studying families of varieties, "moduli spaces", tangent spaces, differential operators, multiplicities etc.
A problem is that when you localize such a ring, it is no longer finitely generated, hence localization "takes you out of the category of Hilbert-Jacobson rings". For this reason you include prime ideals - then the theory becomes "more streamlined".
Title-question: "Why do we need prime ideals in the spectrum of a ring?"
Answer: Below you will find a link to a construction of the functor $F^m$. The functor exists and gives a ringed topological space valid for any "scheme of finite type over a Dedekind domain". When $k$ is a Dedekind domain you do not need prime ideals - the above construction can be used. If $S:=Spec(A)$ where $A$ is a Dedekind domain and if $\pi: X \rightarrow S$ is a scheme of finite type over $S$, you may construct the locally ringed space $(X^m, \mathcal{O}_{X^m})$ where $i:X^m \rightarrow X$ is the subspace (with the induced topology) of closed points and $\mathcal{O}_{X^m}:=i^{-1}(\mathcal{O}_X)$ is the topological inverse image. It follows the locally ringed space $(X^m, \mathcal{O}_X{^m})$ has nilpotent elements in its structure sheaf. The sheaf $\mathcal{O}_{X^m}$ is a sheaf of non-reduced rings in general. It has the property that if $U \subseteq X$ is an open affine set and $U^m:=U \cap X^m$ it follows $\mathcal{O}_{X^m}(U^m) \cong \mathcal{O}_X(U)$. Hence the sheaves $\mathcal{O}_{X^m}$ and $\mathcal{O}_X$ have "the same sections" (in the above sense)  over affine open subschemes.
https://mathoverflow.net/questions/377922/building-algebraic-geometry-without-prime-ideals/378961#378961
Note: I chose to delete my profile at MO and it seems this contribution was deleted by the administrators at MO. My other contributions to MO are filed under "user122276".
