May as well write an answer since part (a) has been done in the comments and (b) is a natural consequence of (a).
Let's give the map in part (a) a name: $\phi : R \times S \to RS$, where $\phi(r,s) = rs$.
We first show that $\phi$ is an injection. Suppose $r_1 s_1 = r_2 s_2$. Then $r_2^{-1} r_1 = s_2 s_1^{-1}$. Since $s_1$ and $s_2$ are in $S$, which is a subset of $H$, and $H$ is a subgroup, it follows that $s_2 s_1^{-1} \in H$, so $r_2^{-1} r_1 \in H$. This means that $r_1 H = r_2H$, in other words $r_1$ and $r_2$ are in the same coset of $H$. Since $R$ contains exactly one representative from each coset of $H$, we must have $r_1 = r_2$. But then $r_2^{-1} r_1 = 1$, the identity element, and this means that $s_2 s_1^{-1}$ is also the identity, so $s_1 = s_2$.
Now we show that $\phi$ is a surjection. This is immediate from the definition, because an arbitrary element of $RS$ is of the form $rs$, where $r \in R$ and $s \in S$, and therefore $rs$ is the image under $\phi$ of the element $(r,s) \in R \times S$.
We conclude that $\phi$ is a bijection.
For part (b), define $\psi : R \times S \to G/K$ by $\psi(r,s) = rsK$.
We show that $\psi$ is an injection. Suppose that $r_1 s_1 K = r_2 s_2 K$. Since $s_1 K \subset H$ and $s_2 K \subset H$, it follows that $r_1 s_1 K \subset r_1 H$ and $r_2 s_2 K \subset r_2 H$. Therefore, $r_1 H$ and $r_2 H$ are not disjoint, so they must be equal: $r_1 H = r_2 H$. This implies that $r_1 = r_2$ by definition of $R$. Let us set $r = r_1 = r_2$. Then we have $rs_1 K = rs_2 K$, so $rs_1 \in rs_2 K$, and therefore $s_1 \in s_2 K$, which means that $s_1 K = s_2 K$. By definition of $S$, this means that $s_1 = s_2$.
We show that $\psi$ is a surjection. If $g$ is an arbitrary element of $G$, then it lies in some element of $G/H$, say $g \in rH$. Then $r^{-1}g \in H$, so $r^{-1}g$ lies in some element of $H/K$, say $r^{-1}g \in sK$. Then $g \in rsK$, so $gK = rsK = \psi(r,s)$.
Finally, note that the statement "$RS$ is a system of representatives of $G/K$" is equivalent to the statement "there is a bijection from $RS$ to $G/K$ which maps $rs \to rsK$". But this is now clear: $\psi \circ \phi^{-1} : RS \to G/K$ is the desired bijection.
Edit to address the question raised in the comment and in the edited version of the question.
We want to conclude that $[G:K] = [G:H][H:K]$.
Note that this follows immediately from what we have already done.
Let's assume that $[G:K]$ is finite. Then, by definition, $[G:K]$ is the number of elements in $G/K$. We have shown that there is a bijection between $G/K$ and $RS$, so $[G:K]$ is also equal to the number of elements in $RS$, which we may denote as $|RS|$. Similarly, $[G:H] = |G/H| = |R|$ and $[H:K] = |H/K| = |S|$. So the equation $[G:K] = [G:H][H:K]$ is the same as $|RS| = |R| |S|$. This in turn is true because we have a bijection between $RS$ and $R \times S$, and the size of $RS$ is $|RS|$, and the size of $R \times S$ is $|R| |S|$.
If $[G:K]$ is infinite, then one or both of $[G:H]$ or $[H:K]$ must also be infinite, again because of the bijection between $RS$ and $R \times S$. So the equation holds (trivially) in that case as well: both sides are $\infty$.