There are two categories you might be thinking of.
1) The category Rel of sets and relations, with composition defined via:
$z \in (P \circ Q)(x) \Leftrightarrow \exists y \in Q(x) \ z \in P(y)$
This category is described in Giorgio Mossa's answer.
2) The category Mult of sets and multivalued functions, with composition defined via:
$z \in (P \circ Q)(x) \Leftrightarrow [ (\forall y \in Q(x) \ P(y) \neq \emptyset) \wedge (\exists y \in Q(x) \ z \in P(y))]$
The motivation of this slightly more complicated definition comes from computer science. There we might want to use multivalued functions to describe (robust) non-deterministic computations - we don't know what output we're going to get precisely, but we know that we are going to get something according to the specification. As the question whether or not our computation will be succesfull is not supposed to depend on the non-deterministic part, we don't consider input valid, if not all outputs of the first multifunction can be put into the second.
A more formal way of expressing this is to look at partial choice functions: If $P : X \to Y$ is a multivalued function, then a partial function $f : \subseteq X \to Y$ is a choice function for $P$ if $P(x) \neq \emptyset$ implies that $f(x)$ is defined and $f(x) \in P(x)$. Now compositions of choice functions are choice functions of the (multivalued) composition, but not necessarily of the relational composition.
Some properties of Mult: Coproducts are just inherited from Set, but products don't work well. In many ways, the category of multivalued functions should be considered as having some addiotional structure, in particular as being poset-enriched.
Reference: http://arxiv.org/abs/1102.3151