Are "$n$ by $n$ matrices with rank $k$" an affine algebraic variety? Identify the set of all complex $n$ by $n$ matrices with $\mathbb{C}^{n^2}$.  We say a subset $S \subset \mathbb{C}^{n^2}$ is an affine algebraic variety if $S$ is the common zero set of a collection (possibly infinite or uncountable) of polynomials in $n^2$ complex variables.  Some examples that satisfy this definition are "matrices with determinant 1" and "matrices with rank at most $k$".  
A non-example is "matrices with rank $n$", because it is the complement of the affine algebraic variety "matrices with rank at most $n - 1$"; any affine algebraic variety is necessarily closed in the Euclidean topology, so its complement (which is open in the Euclidean topology) cannot be an affine algebraic variety.
However, I'm having trouble formulating a statement about "matrices with rank $k$" for $1 < k < n$.

Is the set of complex $n$ by $n$ matrices of rank $k$, where $1 < k < n$, an affine algebraic variety?

 A: No (unless $k=0$). For example this set is not closed: consider diagonal matrices with exactly $k$ non zero entries and make one of these entries go to zero.
A: Well, first I disagree with your definition of an affine algebraic variety.  What you have actually defined is a Zariski-closed subset of affine space.  But sometimes for instance a Zariski-open subset of affine space can be polynomially mapped in a bijective fashion to a Zariski-closed subset of a different affine space.  In particular, here when $k = n$ you are saying that the collection of all invertible matrices is not an affine algebraic variety.  But it is: you just have to embed it into an affine space of one more dimension.  (It is amazing how difficult is for me to avoid saying the word "scheme" here.  I had better press on before I slip up.)
(Added: while it's technically true that the definition of a Zariski-closed subset allows for arbitrary intersections of zero sets of polynomials, the Hilbert Basis Theorem implies that in fact every such set is an intersection of zero sets of finitely many polynomials.  So emphasizing the infinitude of the intersection is perhaps slightly misleading.)
Anyway, you are asking whether the set of matrices of rank $k$ forms a Zariski closed subset.  The answer is no for any $k > 0$.  The point is that the set of matrices of rank at most k forms a Zariski-closed subset: it is defined by the vanishing of all determinants of $(k+1) \times (k+1)$ minors of the matrix.  So the set of matrices of rank exactly $k$ is naturally a "stratum": it is the complement in the closed set of matrices of rank at most $k$ by the set of matrices of rank at most $k-1$.  In general, taking a closed set and subtracting a smaller closed set leaves you with something which is neither open nor closed.  (There is a word for such slightly more complicated than open or closed sets: constructible.)  This is the case here, although that requires some additional argument.  From a geometric perspective, the most natural thing to do is show that the subset of all matrices of rank at most $k$ is not just closed but irreducible: then any relatively open subset must be dense in it, so cannot be closed.  Unfortunately I don't have a reference for this off-hand, but it can probably be found in Joe Harris's GTM on algebraic geometry.  Well, once I looked for a reference I found one here.
Let me say that Didier's argument (which I saw after I started typing this) is a very nice, easy way to answer your question.  He is arguing (I think) with respect to the usual Euclidean topology, but since that is finer than the Zariski topology, if your set is not closed with respect to the Euclidean topology, it is certainly not closed with respect to the Zariski topology.  Anyway, this argument can be adapted to work over any infinite field $K$ and I'll ask you to do it as an exercise: for any $0 < k \leq n$, find a one dimensional linear subspace inside $M_n(K)$ all of whose nonzero points give rank $k$ matrices but for which the origin is (of course) the zero matrix.  Any polynomial function which vanishes at all but one point of this line has to vanish at the other point too!
Added: perhaps some actual algebraic geometer can tell me whether these quasi-affine algebraic varieties actually are affine for $k < n$ the way they are for $k = n$.  I would guess no -- we are removing too little at each step for this to be the case -- but it's just a guess.  I'm sure there must be some nice way to think about this...
A: At the end of his answer, Pete Clark asks whether the quasi-affine variety of $n \times n$ matrices of rank $k$ might be affine. The answer is no, except when $k=n$.
This is a special case of a much more general fact. Let $X$ be an irreducible affine variety, and let $Y$ be a closed subvariety. If $X \setminus Y$ is affine, then $Y$ is  codimension $1$ in $X$.
Proof Sketch: (This argument is edited in response to points raised to me by Steven Sam.)
Let $r$ be the codimension of some component of $Y$. 
Then the local cohomology group $H^r_Y(X, \mathcal{O})$ is nonzero. 
I earlier said that this was by the inequality between depth and codimension, but Steven points out to me that I am misusing this; that inequality implies that $H^s_Y(X, \mathcal{O})$ is nonzero for some $s \leq r$. I think the inequality I meant to cite is Grothendieck's nonvanishing theorem.
Look at Hartshorne Exercise III.2.2.3.e . There is an exact sequence which, in part, is
$$H^r(X \setminus Y, \mathcal{O}) \to H^{r+1}_Y(X, \mathcal{O}) \to H^{r+1}(X, \mathcal{O}).$$
Since $X$ is affine, the right hand side is zero, and we just claimed that the middle term is nonzero. So $H^r(X \setminus Y, \mathcal{O})$ is nonzero. If $r>1$, this implies that $X \setminus Y$ is non-affine. QED
The rank $k$ matrices have dimension $n^2 - (n-k)^2$. One can check that this is only codimension $1$ for $(k, k-1) = (n,n-1)$.
By the way, it is NOT true that $X$ affine, $Y$ codimension $1$ implies $X \setminus Y$ affine. The simplest counter-example I know is $X = \mathrm{Spec}\ k[w,x,y,z]/(wz-xy)$ and $Y = \{ w=x=0 \}$. 
