In what way does replacing maximal ideals with prime ideals yield, intuitively, the theory of schemes? Background: I've only so far seen the most basic aspects of algebraic geometry from Milne's notes on algebraic varieties, and thus my view of the subject as a whole is quite limited. Thus, when answering this question please try and be somewhat concrete or else explain what you're talking about (I have no idea what words like "cohomology" or "derived functor" have to do with AG :-).
The most abstract way of defining an affine variety is to define it as a $k$-ringed space isomorphic to the maximal spectrum $\operatorname{Spm}(A)$ of a (reduced, finitely generated as a $k$-algebra) ring $A$. The usual theory of algebraic sets with preferred embeddings in some affine space $\mathbb A^n$ is the motivating example - namely, take $A=k[V]$ to be the coordinate ring $k[X_1,\dots,X_n]/I(V)$ of some algebraic set $V\subset\mathbb A^n$. Then maximal proper ideals of $k[V]$ correspond to minimal nonempty algebraic subsets - points - and this gives a very intuitive and "classically geometric" reason to generalize to the case when $A$ is not a quotient of a polynomial ring.
What I do know about affine schemes are their definition - they are $k$-ringed spaces isomorphic to the prime spectrum $\operatorname{Spec}(A)$ of some ring $A$. So the difference between schemes and varieties highlights a simple generalization: maximal ideals become prime ideals. This doesn't gel with the geometric theory from before: for instance, the simple case of $\operatorname{Spec}k[X,Y]$ corresponds to the set of irreducible algebraic subsets which doesn't have nearly as immediate a geometric motivation. I understand how allowing nilpotents can be helpful, but can someone explain to me, in simple terms, why this generalization to prime spectra is so good and important?
 A: I'll tell you how it's been explained to me, though I'm sure the story has been sanitized a bit for my consumption.  Any mistakes below are purely my own.

The theory of representable functors is much nicer than the theory of arbitrary functors, so any time we've got a functor we'd like to know if it's representable and, if so, to make use of this.
Now the very close relation between affine varieties and their coordinate rings means that the two classes basically represent the same functors.  For instance, the variety consisting of a single point and its coordinate ring $k$ both represent the functor giving the $k$-points of a variety defined over $k$.
Now, you can have a functor represented by an arbitrary ring, and this is quite useful: for instance, the ring of dual numbers $k[\varepsilon] := k[t]/(t^2)$, which is nonreduced, represents the tangent bundle functor.
On the other hand, a functor can be represented by a non-affine variety, and this is quite useful: for instance, the Grassmannian variety $\operatorname{Gr}(k,n)$ represents the functor giving rank-$k$ subbundles of the rank-$n$ trivial bundle.
Sometimes, though, you'd have a functor that seems like you ought to be able to represent it, except that the thing you need to represent it would somehow be  "both non-reduced and non-affine."  (I think the motivating example here was the Hilbert scheme).  Rings can do the non-reduced part, but not the non-affine part, and varieties can do the non-affine part but not the non-reduced part, so it looks like we're stuck.

Instead of giving up, though, you can just invent schemes, which can be non-reduced and non-affine at the same time.  When you try to synthesize the two theories, though, you run into a bit of a problem: remember the functor of $k$-points?  If $V$ is an affine variety in $k^n$, then
$$\operatorname{Hom}(\{\cdot\}, V)$$
gives us the points of $V$ in the old sense, or in other words the maximal ideals of $k[V]$, whereas
$$\operatorname{Hom}(k[V], k)$$
gives us the prime ideals of $k[V]$.  If we're going to generalize both theories simultaneously, we're going to have to resolve this discrepancy somehow, and since a maximal ideal is a kind of prime ideal there turns out to be a very nice solution.
(Of course the technical issue underlying this is a special case of what Roland said in the comments when we take $B = k$.)

This also aligns quite fortunately with the fact that people such as Emmy Noether and Krull had previously considered (what we'd now call) affine schemes as a way of formalizing arguments from the Italian school which depended on the properties of "a generic point of a variety."  Apparently these ideas hadn't really been taken seriously until it was shown that such a notion was useful in a broader context.
