How to estimate CTR (click-through rate)? How many times banner should be shown to estimate CTR? 
For example, a banner was shown x times, and was clicked y times.
CTR = y/x;

How to evaluate incaccuracy of this value?
 A: As a rule of thumb, errors for a number $y$ have the order of magnitude $\sqrt y$. Thus you may be relatively confident that the "true" CTR (i.e. what can be expected as $x$ becaomes larger over time) is probably between $\frac{y-\sqrt y}x$ and $\frac{y+\sqrt y}x$
A: Here's one way to do that. Assuming that there's a probability $p$ with which a user clicks on a banner that he sees. Then if $n$ banners are shown, the probability of there being $k$ clicks is $$
  p(k) = \binom{n}{k}p^k(1-p)^{(n-k)} \text{.}
$$
To derive this formula, observe that to observe $k$ clicks, the event CLICKED which has probability $p$ has to occur $k$ times, and the event $NOT(CLICKED)$ which has probability $(1-p)$ must occur the other $n-k$ times. The factor $\binom{n}{k}$ is required because you do not care about the precise sequence of events, i.e. about whether $k$ clicks all occur in a row, followed by $n-k$ not-clicks, or are interleaved in some way.
From this, you get that the probability of observing at least $k$ clicks out of $n$ views given $p$ is $$
  P(K \geq k|p) = \sum_{i=k}^n \binom{n}{k}p^k(1-p)^{(n-k)}
$$
and similarly the the probability of observing at most $k$ clicks is $$
  P(K \leq k|p) = \sum_{i=1}^k \binom{n}{k}p^k(1-p)^{(n-k)}
$$
You can now pick an $\alpha \in [0,1]$ and find all $p$ for which $$
  P(K \leq k|p) > \frac{\alpha}{2} \text{ and } P(K \geq k|p) > \frac{\alpha}{2} \text{.}
$$
The idea being basically that you rule out all values of $p$ for which the probability of observing either fewer or more than $k$ clicks out of $n$ views is larger than $1-\frac{\alpha}{2}$. Thus, the smaller you make $\alpha$, the fewer values of $p$ you rule out. This, btw, is called the Clopper-Pearson interval.
