I have a set of functions that map square matrices with $n$ rows and columns to square matrices with $k < n$ rows and columns. Is there a name for this property? I know that 'projection' would be the wrong word. Also I don't mean that the function only changes the size, it might also do other things.
Well, projection could be the right word, depending on what you want to do with the matrices. Note that matrices with $n$ rows and $m$ columns form a vector space of dimension $n \times m$. If you now restrict to matrices of $k < n$ rows, you can treat them as a $k \times m$-dimensional subspace of the original vector space, having entries in rows $i > k$ set to zero.
Also note that the dual operation of reducing the number columns might be called a restriction. This comes from the fact that you can treat $n \times m$ dimensional matrix as a representation of an operator mapping $m$-dimensional vector space to a $n$-dimensional vector space and reducing number of columns restricts this operator to a subspace of the original vector space.
If the space of nxn and kxk matrices are viewed as vector spaces, then functions mapping between them would be called "operators". That's probably not as specific as you want since it doesn't take into account the matrix structure, only the vector space structure.