analogy between iid and sampling with replacement Are iid samples and sampling with replacement analogous to each other?
iid - Independent and Identically Distributed RVs, means that each sample is independent of all others and all samples are drawn from the same distribution.
sampling with replacement - You sample a value from a distribution ( eg. A ball from an urn) and once you note it down, you replace it back. Does replacement ensure that the draws are independent and from identical distributions?
I have not seen this analogy spoken about in statistics books, maybe it's trivial but I wanted to make sure my intuition was correct.
 A: Yes, in sampling with replacement with all outcomes equally likely, the samples are indeed independent and identically distributed.  
Suppose you have a bin with $n$ balls, numbered $\{1, ..., n\}$, and you sample 3 times with replacement.  So all outcomes have the form $(x_1, x_2, x_3)$, where $x_i$ represents the ball number of sample $i \in \{1, 2, 3\}$, and $x_1, x_2, x_3 \in \{1, ..., n\}$.  So there are a total of $n^3$ possible outcomes.  Suppose all $n^3$ outcomes are equally likely. Then each outcome has probability $1/n^3$. Since an event is just a set of outcomes, the probability of any event $B$ is equal to:
$$ P[B] = \frac{|B|}{n^3} $$
where $|B|$ denotes the number of outcomes in the event $B$.
Define the random variable $X_i$ to be the ball number of sample $i$. 
For given integers $k_1, k_2, k_3 \in \{1, ..., n\}$, define the events: 
\begin{align}
A_1 &= \{X_1 \leq k_1\}\\
A_2 &=\{X_2 \leq k_2 \}\\
A_3 &= \{X_3 \leq k_3\}
\end{align}
Then $A_1 \cap A_2 \cap A_3$ is the event that all three events happen.  There are $k_1k_2k_3$ ways for this to occur, that is,  $|A_1 \cap A_2 \cap A_3|=k_1k_2k_3$.  So: 
$$P[A_1 \cap A_2 \cap A_3] = \frac{k_1k_2k_3}{n^3}=\frac{k_1}{n}\frac{k_2}{n}\frac{k_3}{n}$$
On the other hand, you can show that $|A_1|=k_1n^2$, $|A_2|=k_2n^2$, $|A_3|=k_3n^2$, and so
$$P[A_1]=\frac{k_1}{n}, P[A_2]=\frac{k_2}{n}, P[A_3]=\frac{k_3}{n} $$
Thus, 
$$P[A_1 \cap A_2 \cap A_3] \overset{(a)}{=} P[A_1]P[A_2]P[A_3]$$
Similarly, you can show 
\begin{align}
P[A_1\cap A_2] &\overset{(b)}{=}P[A_1]P[A_2]\\
P[A_1\cap A_3] &\overset{(c)}{=}P[A_1]P[A_3]\\
 P[A_2\cap A_3]&\overset{(d)}{=}P[A_2]P[A_3]
\end{align} 
We deduced all this directly from the fact that all samples are equally likely, without using the concept of independence.  However, this motivates a good definition for independence.  Indeed, three events $A_1, A_2, A_3$ are defined to be mutually independent if equalities (a)-(d) hold. 
In terms of the random variables $X_i$, the equality (a) implies: 
$$P[X_1\leq k_1, X_2\leq k_2, X_3 \leq k_3]\overset{(e)}{=}P[X_1\leq k_1]P[X_2\leq k_2]P[X_3\leq k_3]$$
This holds for all integers $k_1, k_2, k_3 \in \{1, ..., n\}$. With some thought, you can convince yourself that it also holds for all real numbers $k_1, k_2, k_3 \in \mathbb{R}$.  In general, we define three random variables $X_1, X_2, X_3$ to be mutually independent if equality (e) holds for all real numbers $k_1, k_2, k_3 \in \mathbb{R}$. Hence, the three random variables  $X_1, X_2,X_3$ of interest here are mutually independent. In fact, it is easy to see that they are also identically distributed.  
