A biased coin is tossed. The probability of heads turning up is 0.51.
Suppose you keep tossing the coin multiple times and keep a track record of the number of heads/tails. The moment you get more heads than tails, you stop tossing the coin.
Mathematically, can we safely say that for all series of coin tosses, the ultimate result is that more heads have turned up and hence the probability of more heads turning up ultimately is tosses is 1?
This question is related to the well-studied problem of random walks. If we let $X_n$ be the difference between the number of heads seen in the first $n$ tosses and the number of tails seen in the first $n$ tosses, positive if there are more heads than tails and negative if there are more tails than heads, and set $X_0 = 0,$ then the sequence of variables $X_0, X_1, X_2, \ldots$ form a random walk over the integers with a $0.51$ chance of moving one place to the right on each move and a $0.49$ chance of moving to the left.
The question is (in effect) asking whether this random walk will eventually reach the value $K_k = 1$ (for some $k$) with probability $1.$ The answer is yes.
A less obvious fact is that the answer is still "yes" even if we replace the biased coin with a fair coin.
Just to complement on the answers given, this actually follows from the Law of Big Numbers: for our example, we have a collection of Bernoulli trials $$X_1,\ldots ,X_m$$ with $X_i\in\{0,1\}$ (1 for heads and 0 for tails) and such that $\mu_i=E(X_i)=0.51$ for each $i$. If we define $$\hat X_m=\frac{1}{m}\,\sum_{i=1}^m\,X_i$$ then, following the (Weak) Law of Big Numbers, we will have that $$\lim_{m\to\infty}\,P(|\hat{X}_m-\mu|\le\epsilon)=1$$ for all $\epsilon>0$. The Strong Law of Big Numbers goes a bit further and claims that $$P(\lim_{m\to \infty}\hat{X}_m=\mu)=1$$
Thus, it's not only that we would always be able to toss the coin enough times so that the number of heads (1's) were bigger than the number of tails (0's) but that we can theoretically keep tossing the coin to get as close as desired to 0.51 as the ratio of heads over total tosses.
