Deriving the probability of an event from limited trial data I have a stochastic event (say a weighted coin toss) that produces a positive outcome (heads) according to some unknown probability $P$.
Given $N$ total events (coin tosses) and n positive outcomes (heads), how can I measure the likelihood that $\frac nN$ is a good approximation of $P$?
Obviously, as $N$ grows, it becomes more likely than $\frac nN$ is close to the value $P$, but how can this convergence be described mathematically?
 A: While I realise this isn't a full answer, the convergence you discuss is guaranteed by the weak and strong laws of large numbers. You might also be interested in Chebychev's inequality, which would say $$\mathbb{P}(|n/N - p| > \alpha) \leq \frac{\sigma^2}{\alpha^2}$$ where $\sigma^2$ is the variance of your estimator, $n/N$. Finally the Central Limit Theorem (CLT) proves that as $N \to \infty$ the estimator becomes normally distributed about $p$ with variance $\frac{\sigma^2}{N}$. 
A: This is the problem of confidence intervals in statistics.  Your question is about the original problem of that type, the  "confidence interval for a proportion in binomial sampling" (wikipedia http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval ) 
The asymptotic theory, when $Np(1-p)$ is very large, is described by the Central Limit Theorem, which says that $n = pN + A\sqrt{Np(1-p)}$ where the probability distribution of $A$ is the famous Gaussian (normal, "bell shaped") curve.  Qualitatively, fluctuations of $n$ around the expected value are usually within a small factor of $\sqrt{Np(1-p)}$.   
For limited data, the story is more complicated, with no perfect solution, many proposals, and a large literature.
A: The wiki on convergence of random variables is what you probably want to read. It discusses various forms of convergence and in particular convergence in probability and almost sure convergence are what you need to take a look at.
