why is this composition well defined? Let's begin with a definition: a class of morphisms $S\subset\mathrm{Mor}\;B$ (where $B$ is a category) is said to be localizing if the following conditions are satisfied:
a) $S$ is closed under composition: $id_X\in S$ for any $X\in\mathrm{Ob}\;B$ and $st\in S$ for any $s,t\in S$ whenever the composition is defined.
b) Extension conditions: for any $f\in\mathrm{Mor}\;B$, $s\in S$ ($f:X\rightarrow Y$, $s:Z\rightarrow Y$) there exist $g\in\mathrm{Mor}\;B$, $t\in S$ ($g:W\rightarrow Z$, $t:W\rightarrow X$) such that $ft=sg$ and if the arrows are in the other direction we have $tf=gs$.
c) Let $f,g$ be two morphisms from $X$ to $Y$; the existence of $s\in S$ with $sf=sg$ is equivalent to the existence of $t\in S$ with $ft=gt$.
So suppose that $S$ is localizing.
Now a roof is a pair $(s,f)$ with $s\in S$ and $f\in\mathrm{Mor}\;B$, $s:X^\prime\rightarrow X$ and $f:X^\prime\rightarrow Y$.
Suppose we have a roof $(s,f)$ $s:X^\prime\rightarrow X$ and $f:X^\prime\rightarrow Y$, and a second roof $(t,g)$ $t:X^{\prime\prime}\rightarrow X$ and $g:X^{\prime\prime}\rightarrow Y$. We say that these two roofs are equivalent if there exists a third roof $(r,h)$ $r:X^{\prime\prime\prime}\rightarrow X^\prime$, $h:X^{\prime\prime\prime}\rightarrow X^{\prime\prime}$ such that $sr=th$ and $fr=gh$.
If we have two roofs $(s,f)$ and $(t,g)$ ($s:X^\prime\rightarrow X, f:X^\prime\rightarrow Y, t:Y^\prime\rightarrow Y, g:Y^\prime\rightarrow Z$) we can compose them. First of all we use property b of the localizing class of morphisms $S$ on the morphisms $f,t$ to get $t^\prime:X^{\prime\prime}\rightarrow X^{\prime}$ and $f^\prime:X^{\prime\prime}\rightarrow Y^\prime$, such that $ft^\prime=tf^\prime$. So the composition of $(s,f)$ and $(t,g)$ is by definition $(st^\prime, gf^\prime)$.
Could you help me to prove that this composition is well-defined? I.e. it doesn't depend on the choice of the representative of the equivalence classes?
I tried but things get really messy
 A: Today I don't have the time to think about your proof, but I would like to share some heuristics about roofs. Maybe they could be helpful.
These roofs $(s,f)$ (or its classes modulo the equivalence relation you have recalled) are the morphisms of the localised category $S^{-1}B$, or $B[S^{-1}]$. The objects of this category are those of $B$ and morphisms those classes we are talking about, but what we actually want is just add to morphisms of $B$ the "inverses" of morphisms in $S$ -even if they are not isomorphisms in $B$.
I think there are two ways of "visualising" these morphisms of $S^{-1}B$. First, recalling that (the class of) a roof such $(s,f)$ in $S^{-1}B$ is a morphism from $X$ to $Y$:
$$
X \stackrel{s}{\longleftarrow} X' \stackrel{f}{\longrightarrow} Y
$$
because $s$ in $S^{-1}B$ is an isomorphism. So, you could also write it as $fs^{-1} : X \longrightarrow Y$ in $S^{-1}B$. Or, even more appealingly, as
$$
\dfrac{f}{s} \ .
$$
Unfortunatelly, I don't have with me the book by Gelfand and Manin, but I suspect there is something missing and could be the following: in this situation, we don't ask $S$ just to be closed by composition, but to have the two out of three property, which says that whenever you have a composition like $u = th$ in which two of the morphisms are in $S$, then so is the third. So, for instance, if $u,t \in S$, then also $h \in S$. This two out of three property is always verified by classes $S$ defined as those morphisms which become isomorphisms after applying some functor (some homology functor, for instance).
So, in this case, the equivalence relation of roofs is indeed symmetrical: since $sr = th$ and $s, r, t \in S$, we also have $h\in S$.
This long detour wants to get to the following heuristics: the equivalence relation of roofs is just
$$
\dfrac{f}{s} = \dfrac{fr}{sr} = \dfrac{gh}{th} = \dfrac{g}{t} \ ,
$$
because you can "cancel out" invertible guys like $r$ and $h$, can't you?   :-)
A: I think this (good) exercise in the use of the calculus of fractions is made a bit more difficult than necessary because Gel'fand–Manin formulate the axioms in a self-dual way. Since we're dealing with (what I call) right fractions, it seems like a good idea to isolate the necessary parts of the axioms for this exercise.  That this is possible is hinted at in remark 9 following the lemma you're trying to understand. Proofs sometimes become easier when one reduces the options one has and this is one instance, I believe.
To get to the maths, the axioms for right fractions read:

Let $\mathscr{C}$ be a category. A collection $\Sigma$ of morphisms in $\mathscr{C}$ is called a right multiplicative system (or right denominator set) if
  
  
*
  
*[RF 0] (non-triviality): the identity morphism of each object belongs to $\Sigma$: for all objects $C \in \mathscr{C}$ we have $1_C \in \Sigma$.
  
*[RF 1] (composition): if $s\colon A \to A'$ and $s'\colon A' \to A''$ are composable morphisms from $\Sigma$ then $s's\colon A \to A''$ belongs to $\Sigma$, too.
  
*[RF 2] (Ore condition): Given an arbitrary morphism $f\colon A \to B$ and a morphism $t\colon B' \to B$ from $\Sigma$, there exists a commutative square

with $s\colon A' \to A$ from $\Sigma$. Note: If $f$ happens to be from $\Sigma$ we do not assume that we can choose $f'$ to be from $\Sigma$, however, we can say that the composition $sf' = tf$ belongs to $\Sigma$ by [RF 1] because both $t$ and $f$ do.
  
*[RF 3] (cancellation condition): If $f,g\colon A \rightrightarrows B$ are two parallel morphisms such that there is $s\colon B \to B'$ from $\Sigma$ such that $sf = sg$, then there is $t\colon A' \to A$ from $\Sigma$ such that $ft = gt$.
  

I use the notation $(f/s)$ with $s \in \Sigma$ to represent a right fraction $A \xleftarrow{s} A' \xrightarrow{f} B$ (a “left $\Sigma$-roof” in Gel'fand–Manin's terminology): my rationale for the name-switching in the handedness is that $(f/s)$ represents “$f \circ s^{-1}$” in the localized category $\mathscr{C}[\Sigma^{-1}]$, so we compose $f$ with $s^{-1}$ on the right to get a morphism $A \to B$ in $\mathscr{C}[\Sigma^{-1}]$.
The above axioms are enough to prove that equivalence of (right) fractions is indeed an equivalence relation. The definition of equivalence of fractions as phrased in Gel'fand–Manin is somewhat confusing because it may suggest an incorrect interpretation (as witnessed in your question). Here's the diagram defining equivalence between $(f_1/s_1)$ and $(f_2/s_2)$ that is actually intended: 

Note that the vertical morphisms $\bar r_1$ and $\bar r_2$ in this diagram are denoted by $r$ and $h$ by Gel'fand–Manin and neither of them is assumed to belong to $\Sigma$. What is assumed to be in $\sigma$ is the composition $\bar{s} = s_1\bar{r}_1 = s_2\bar{r}_2$.
If you read the proof of Lemma 8 a) of Gel'fand–Manin with this definition in mind, you'll see that they actually prove that the above relation is transitive (only using the parts of their axioms that I gave above), so I'll take that result for granted.
Before proceeding further, a trivial but useful

Remark. The equivalence diagram not only tells us that $(f_1/s\vphantom{f}_1)$ and $(f_2/s\vphantom{f}_2)$ are equivalent, but that they both are equivalent to their common expansion $(\bar f/\bar s)$: the upper half exhibits $(\bar{f}/\bar{s})$ to be equivalent to $(f_1/s\vphantom{f}_1)$ and the lower half gives that $(\bar{f}/\bar{s})$ is equivalent to $(f_2/s\vphantom{f}_2)$.


I think it is instructive to break your question into small and easy steps instead of proving the general well-definedness in one blow, so that's the route I'll take below. I'll give the details of most of the argument and think that it is safe to leave the rest only in outline. The direct route is of course possible, but, as you mentioned, it becomes somewhat messy.
One crucial observation to make towards proving the well-definedness of composition in the category of fractions is that the Ore condition [RF 2] reads $fs = tf'$, or “$t^{-1} \circ f = f' \circ s^{-1}$”, so it is not too surprising that we have the following:

Lemma. Given a wedge $A \xrightarrow{f} B \xleftarrow{t} B'$ in $\mathscr{C}$ with $t \in \Sigma$, any two right  fractions $(f_1/s_1)$ and $(f_2/s_2)$ such that $tf_1 = fs_1$ and $tf_2 = fs_2$ are equivalent: “up to equivalence of fractions there's only one Ore square over any given wedge”.

(This observation is a variant of the heuristics at the beginning of remark 7 in Gel'fand–Manin)
Proof. We are given the two commutative squares

and we apply the Ore condition to the wedge $A_1 \xrightarrow{s_1} A \xleftarrow{s_2} A_2$ to get the commutative square

Observe that $s := s_1s_1' = s_2s_2' \in \Sigma$ because of the composition axiom. Moreover, we have
$$
t f_1 s_1' = f s_1 s_1' = f s_2 s_2' = t f_2 s_2'
$$
so that there is $s'\colon A'' \to A'$ such that $f_1 s_1' s' = f_2 s_2' s'$ by the cancellation axiom.
But this means that the diagram

is commutative, proving the equivalence of $(f_1/t_1)$ and $(f_2/t_2)$.

Finally, here's how the above leads to the desired conclusion:


*

*In order to compose $(g/t)$ with $(f/s)$ apply the Ore condition to $f$ and $t$ to get the diagram

and apply the lemma to prove that the equivalence class of the composition $(g/t) \circ (f/s)$ is independent of the chosen Ore square over $f$ and $t$.

*Replace $(f/s)$ by an expansion $(f_1/s_1) = (fr/sr)$ to see that the compositions $(g/t) \circ (f/s)$ and $(g/s) \circ (f_1/s_1)$ are equivalent, as shown by the following diagram

(this diagram proves all that you need for this case).

*Similarly, replace $(g/t)$ by an expansion $(g_1/t_1)$ to see that the compositions $(g/t) \circ (f/s)$ and $(g_1/t_1) \circ (f/s)$ are equivalent (I leave this case as an exercise).

*Combine 2. and 3. and use transitivity of equivalence of right fractions to see that the compositions $(g/t) \circ (f/t)$ and $(g_1/t_1) \circ (f_1/s_1)$ are equivalent: $(g/s) \circ (f/s) \sim (g/s) \circ (f_1/s_1)  \sim (g_1/s_1) \circ (f_1/s_1)$, provided $(f_1,s_1)$ and $(g_1/t_1)$ are expansions of $(f/t)$ and $(g/t)$, respectively.

*Use transitivity of equivalence of fractions to see that if $(f/s) \sim (f'/s')$ and $(g/t) \sim (g'/t')$ lead to $(g/t) \circ (f/s) \sim (g'/t') \circ (f'/s')$ by using 4. in order to compare $(f/s)$ and $(f'/s')$ to a common expansion $(f_1/s_1)$ as well as $(g/t)$ and $(g'/t')$ to a common expansion $(g_1/t_1)$.

To finish up, let me direct you to this thread for a discussion of related questions, some remarks on the significance of the “2-out-of-3” condition Agustí mentioned (it is called saturation in the other thread), as well as some useful links and references at the end the answer.
