Sample means and confidence intervals For a population, where its parameters are unknown, why is it that if we take samples, those samples create a distribution that resembles that of a Normal Distribution? What if the original population distribution is not normal, why don't the samples create a distribution that is similar to the population, because they should have similar features to the population?
 A: For any given sample, you are correct - the distribution of values within the sample will resemble the population distribution. For example, if our population is a uniform distribution on $[0, 1]$, then it's reasonable to expect that about 10% of our sample will wind up between 0 and 0.1, another 10% will be between 0.1 and 0.2, and so forth.
However, the Central Limit Theorem isn't about the distribution within the sample, it's about the distribution of the sample mean across all possible samples. To get a feel for how that works, let's have our distribution be $X = Bernoulli(p)$, i.e. $X = 1$ with probability $p$ and $X = 0$ with probability $1 - p$. So our sample is $s = (X_1, X_2, \ldots X_n)$, where $X_i \sim X$ are all independent. Since each of $X_i$ has two possible outcomes, there are $2^n$ possible samples, i.e. there are $2^n$ different "realisations" of $s$, and you could even figure out the various probabilities.
However, if we look at the sample mean $\bar{X} = \frac{1}{n}(X_1 + \ldots + X_n)$, there are only $n+1$ possible values between 0 and 1, which means that multiple possible outcomes for $s$ correspond to the same value of $\bar{X}$, which makes sense since $\bar{X}$ only depends on the count of how many of the $X_i$ are equal to 1 and doesn't care about their specific order.
So what does the distribution of $\bar{X}$ look like? Well, obviously it's $\bar{X} \sim B(n, p)$ i.e. a binomial distribution, but in a less technical sense you can see that it's going to have a very low probability of being exactly equal to 0 or 1 since those both require all of the $X_i$ to take on specific values, and in fact we "expect" that most of the time you're going to get $\bar{X} \approx p$, i.e. the proportion of $X_i$ that are equal to 1 is roughly equal to the probability that any individual one does indeed come out that way. Of course it's probably fairly unlikely that $\bar{X} = p$ exactly (in fact for many values of $p$ and $n$ it won't even be possible), but it seems fair to expect that it will be "close" with high probability, for some value of "close". You might even guess that things become a little more symmetric regardless of the original value of $p$, at least in the sense that $X_i$ being slightly less than $p$ will probably wind up being roughly as likely as $X_i$ being slightly above $p$ by the same amount.
The Central Limit Theorem gives a more exact idea of what actually happens, and what it says is that as $n$ gets large, the distribution of $\bar{X}$ is going to become a better and better approximation of a normal distribution, meaning that you can make some very accurate predictions of what the mean of the sample values will be even if you don't know much about how the individual $X_i$ values might be distributed within the sample.
A: It is not the case that samples from an arbitrary distribution are normal.
What you are thinking of is the Central Limit Theorem (CLT) which instead tells us the following...
Let $F$ be a (sufficiently regular) distribution with mean $\mu$ and standard deviation $\sigma$. Let $X_{1},X_{2},\ldots$ be a sequence of IID samples from $F$.
Then, the sample mean is "roughly" normal when a "large" number of samples $n$ are used:
$$
\frac{X_{1}+\cdots+X_{n}}{n}\approx N(\mu,\sigma^{2}/n).
$$
Note, in particular, that the uncertainty in our estimate of the true mean by the sample mean tends to zero as more and more samples are used (i.e., $n \rightarrow \infty$).
The above is not a rigorous statement ($\approx$).
More rigorous forms of the CLT involving different flavors of convergence can be found on the corresponding Wikipedia article.
I recommend you visit these in the future after learning about convergence of random variables.
A: As others have stated, it is the sample mean, under rather general conditions, that converges to a normal distribution. Specifically, if you are drawing iid samples from a population with mean $\mu$ and variance $\sigma^2$ then
$$\frac{\sum_1^n \left(X_1-\mu\right)}{\sigma \sqrt{n}} \xrightarrow{d} N(0,1)$$
You'll notice that the denominator grows exactly as fast as the standard deviation of the numerator, so it doesn't converge (in probability), but if you draw $M$ samples of size $n$ and calculate the following statistic for each of the $M$ samples:
$$Z_n=\frac{\sum_1^n \left(X_1-\mu\right)}{\sigma \sqrt{n}}$$
You'll see that the empirical distribution (ECDF) will get closer to $\Phi(z)$ the larger $n$ gets, where we can define "closer" using the Kolmorogov-Smrinov distance (d_{KS}) to the standard normal distribution:
$$d_{KS} := \sup_{z} |\Phi(z) - ECDF(z)|$$
Why does this matter? When we are trying to estimate some aspect of a population, perhaps a parameter from a model, it's often the case that you define an estimator (e.g., like our $Z_n$) that converges to a normal distribution as $n\to \infty$ so you have a gauge on how accurate an estimate is likely to be (on average).
