How does one find the density of the $k$th ordered statistic? Let $X_1,\ldots,X_n$ be $n$ iid random variables. Suppose they are arranged in increasing order 
$$X_{(1)}\leq\cdots\leq X_{(n)}$$
The first ordered statistic is always the minimum of the sample
$$Y_1 \equiv X_{(1)}=\min\{\,X_1,\ldots,X_n\,\}$$
For a sample of size $n$, the $n$th order statistic is the maximum, that is,
$$Y_n \equiv X_{(n)}=\max\{\,X_1,\ldots,X_n\,\}$$
According to Wolfram Mathworld (here), if $X$ has a probability density function $f(x)$ and cumulative distribution function $F(x)$, then the probability $Y_r$ is given $$f_{Y_r} = \frac{N!}{(r-1)!(N-r)!} [F(x)]^{r-1} [1-F(x)]^{N-r}f(x)$$
My Questions


*

*Is an ordered statistic merely the ordering of a collection of random variables from highest to lowest? But I thought a random variable was a collection of values (e.g. $\lbrace 1,2,3,4,5,6 \rbrace$), so how can we rank them? 

*How do I derive that formula for $f_{Y_r}$? I googled it several times and I don't get how I found a would be derivation here I think on slide 4, but I don't follow what's going on. 

 A: *

*I believe that that ordered statics are the ordering of realizations of $X_1,\dotsc, X_n$ from lowest to highest. A random variable is a function.

*The thing you posted for $f_{Y_r}$ is the formula for the $r$th order statistic. I do not advocate the memorization of this formula. And for a long time, I didn't even know it existed. To me, it is better to understand what is happening. I provide a simple example with a very well known result. In the first part, I talked through it, but it is essentially the formula you provide. You can see how it would generalize to an arbitrary number $n$, with a generic distribution $G$. In the second part, I approach using the cdf.


Let $Y_1,Y_2,Y_3$ iid exponential distributions with mean $1/\lambda$. Then, to find distribution of the minimum $M:=Y_{(1)}$, we must consider
\begin{align*}
P(M\in dm) &= P(M\in [m,m+\epsilon])\tag 1\\
&=P(\text{one of the $Y$'s } \in [m,m+\epsilon]\text{ and all others }>x)\tag 2
\end{align*}
where in $(1)$ and $(2)$ I use the notation provided in the slides.
Since there are $3$ choices for the minimum


*

*$Y_1$, or

*$Y_2$ or

*$Y_3$,


the number of ways to choose a minimum is 
$$\binom{3}{1} = 3$$
where $\binom{n}{k}$ is the counting factor called the binomial coefficient. Notice that I am not invoking the binomial distribution. In other words, there are $3$ choices to fall in the interval $[m,m+\epsilon]$.
Once we have chosen the smallest, then there is only
$$\binom{2}{2} = 1$$ way to choose the other two larger ones. In other words, there is one way have the rest fall into the interval $(m,\infty)$.
Recall that the $Y_i$ are iid $Y\sim\text{Exp}(\lambda)$, by which I mean that the average is $1/\lambda$.
So it must be the case that
\begin{align*}
P(M\in dm) &= P(M\in [m,m+\epsilon])\\
&=P(\text{one of the $Y$'s } \in [m,m+\epsilon]\text{ and all others }>m)\\
&=P(\text{one of the $Y$'s } \in [m,m+\epsilon])P(\text{All others }>m)\tag 3\\
&=\binom{3}{1}[f_Y(m)\epsilon]\binom{2}{2}[1-F_Y(m)]^2 \\
&=3[\lambda e^{-\lambda m}\epsilon][1-\{1-e^{-\lambda m}\}]^2 \\
&= 3\lambda\epsilon e^{-3\lambda m} 
\end{align*}
where in $(3)$ I use independence.
This gives that the pdf is
$$f_M(m) = 3\lambda e^{-3\lambda m}.$$
Notice that the minimum $M$ (or $Y_{(1)}$) follows an exponential distribution with mean $\frac{1}{3\lambda}$.


Alternatively, for the $n$ case
$$P(Y_{(1)}\leq m) = 1-P(Y_{(1)}>m) =1- (e^{-\lambda m})^n = 1- e^{-n\lambda m}.$$
Then the pdf is $n\lambda e^{-n\lambda m}$. Notice that the interpretation is that
there is $\binom{n}{1}$ options for the smallest. Then the rest $\binom{n-1}{n-1} = 1$ have to fall in the interval $(m,\infty)$, hence $n\lambda e^{-n\lambda m}$. 
A: $\{X_1, ... X_n\}$ is a collection of samples no particular size order; usually arranged by sampling time, location, or whatever.   Eg: $\{0.4817, 0.2318, 0.7875, 0.1115\}$
$\{X_{(1)},...X_{(n)}\}$ is that collection reordered from least to most.   So that example becomes $\{ 0.1115, 0.2318, 0.4817, 0.7875\}$

For a continuous random distribution: formulae for $f_Y(y)$ (actually $f_{X_{(r)}}(y)$ is derived by:

*

*Counting ways to selecting $r-1$ samples to be smaller, $n-r$ samples to be larger, and 1 sample to be the favoured sample out of all ways the samples could be ordered.  $\dfrac{n!}{(r-1)!\,(n-r)!}$

*measuring the probability density of the favoured sample: $f_X(y)$.

*measuring the probability that the $r-1$ samples are less than this $F_X(y)^{r-1}$

*measuring the probability that the $n-r$ samples are more than this $(1-F_X(y))^{n-r}$

*Putting it all together.

$$f_{X_{(r)}}(y) = \dfrac{n!}{(r-1)!\,(n-r)!} F_X(y)^{r-1} (1-F_X(y))^{n-r} f_X(y)$$
Note: this only works then there is a zero probability of ties; that is continuous random variables only.  The probability mass function of order statistics for discrete random variables is much more involved.
A: The probability that the $k$th order statistic is less than $x$ is the probability of the union of mutually exclusive events where at least $j \geq k$ of the samples are less than $x$ but the remaining $n-j$ samples are greater than $x$. Note that there are ${n}\choose{j}$ different combinations satisfying this condition. The probability of this union is a sum of probabilities over $j=k, \ldots,n$.
Assuming an iid sample of size $n$, the distribution function for $X_{(k)}$ is
$$\mathbb{P} (X_{(k)}\leq x) = \sum_{j=k}^{n} {{n}\choose{j}}[F(x)]^j[1-F(x)]^{n-j}$$
To get the density function, take the derivative of the distribution function with respect to $x$ to find
$$f_{(k)}(x) = \frac{n!}{(k-1)!(n-k)!}F(x)^{k-1}(1-F(x))^{n-k}f(x).$$
This a lengthy calculation and I believe there is no easy path other than something along the lines of
https://math.stackexchange.com/a/1179378/148510
