Negative variance. 
I've an urn with 5 red and 3 white balls. I'm randomly drawing a ball
  from the urn each round (without replacement). Let $X$ be the number
  of balls taken from the urn until the first white ball is chosen. I'm
  asked for the pmf, expected value and variance of $X.$

Since we can at most take 5 red balls before there is a white ball with 100 % certainty, we have that $k=1,2,3,4,5.$ We can ignore all the previous balls that we don't look at, the probability will therefore remain the same for all the draws, that is the pdf is $$P(X=k)=3/8, \ \ k=1,2,...,5.$$
For the expected value we then have that 
$$E(X)=\sum_{k=1}^{5}k\cdot\frac{3}{8}=\frac{45}{5}\approx 5.6.$$
For the vairance, we also need $E(X^2),$ so 
$$E(X^2)=\sum_{k=1}^{5}k^2\cdot\frac{3}{8} = \frac{165}{8}\approx 20.6.$$
So $$Var(X)= E(X^2)-E(X)^2=20.6-5.6^2=-10.67.$$
But negative variance, can't be correct! Where am I wrong?
 A: The error stems in the first part of your answer, when providing the probability distribution.
You state that the pdf is $$P(X=k)=3/8, \ \ k=1,2,...,5.$$
Note that this means that the probability $P(X=1)=3/8$ and $P(X=2)=3/8$ and $P(X=3)=3/8$ and so on. For a pdf, the total probability over all values of $k$ should sum to $1$, but from the first $k=1,2,3$ alone, we get that the probability is $9/8$ which is greater than $1$. This means you've miscalculated these probabilities.
To calculate these correctly, let's take your observation that after the 5th ball we're guaranteed to get a white ball. The probability that the first ball is white is: $3/8$, because there are 3 white balls and 8 balls total, so $P(X=1)=3/8$. Now note that the probability that the 2nd ball is white is $\frac{5}{8} * \frac{3}{7}$, because for the 2nd ball to be our first white ball, the first ball we drew must have been red. At that time, there were 5 red balls out of 8 total balls. After our first draw, we wanted to draw a white ball, and we had 3 white balls out of the 7 total balls left (since one red has been drawn), leading us to find that $P(X=2)=\frac{5}{8} * \frac{3}{7}$
Note then that following this pattern, we get that $P(X=6)=\frac{5}{8} * \frac{4}{7} * \frac{3}{6} * \frac{2}{5} * \frac{1}{4} * \frac{3}{3}$. Can you understand why this is? And also, can you tell why $P(X=k)=0$ for any $k > 6$?
To show an example of how to represent this PDF in general form, let's take the case $P(X=3)=\frac{5}{8} * \frac{4}{7} * \frac{3}{6}$
With $k = 3$, balls $1$ and $2$ are going to be red, so like @JMoravitz suggests user falling factorials for this. $\frac{5!}{(5−2)!}$ can represent the denominator and $\frac{8!}{(8−2)!}$ can represent the numerator. Note that the $2$ in both cases comes from the fact that only the first $k - 1$ balls will be red. For the white ball then, we calculate the probability as $\frac{3}{8 - 2}$ where $2$ comes here from the fact that we've drawn $k - 1$ balls for the reds already. So then, $P(X= 3)=\frac{\frac{5!}{(5−2)!}}{\frac{8!}{(8−2)!}} * \frac{3}{8 - 2}$. 
A: Continuing from comments:
Letting $X$ be the random variable counting the number of balls pulled until pulling the first white ball, we try to find the probability distribution.
To do this, temporarily label each of the balls uniquely.  For the $k$'th ball to be the first white ball, that implies that the first $k-1$ balls are all red.  These $k-1$ red balls can be pulled in $5\frac{k-1}{~}=P(5,k-1)=\frac{5!}{(5-(k-1))!}$ ways.  (For example, pulling two red balls can be done in $5\cdot 4$ ways and pulling three red balls can be done in $5\cdot 4\cdot 3$ ways, etc...).  Then, pulling a white ball can be done in $3$ ways, so multiplying we get a total of $5\frac{k-1}{~}\cdot 3$ ways in which we can pull $k$ balls where only the $k$'th ball is white.  Then, recognize that the number of ways of pulling $k$ balls without regards to color will be $8\frac{k}{~}$.
Taking the ratio then yields the probability we are after: $Pr(X=k)=\dfrac{5\frac{k-1}{~}\cdot 3}{8\frac{k}{~}}$ (for values of $k\geq 1$.  For $k=0$ the probability is zero as pulling a white ball requires pulling at least one ball)
Alternatively, one could see the same result by using conditional probability arguments.  For the $k$'th ball to be the first white ball to occur, that implies that the first ball is red which occurs with probability $\frac{5}{8}$, the second ball is red which occurs with probability $\frac{4}{7}$ given that the first ball was red, the third ball was red... on up until the $k-1$'st ball is red, followed by the final ball being white.
Rewritten and evaluated, this yields the following table:
$\begin{array}{|c|c|c|c|}\hline k&Pr(X=k)&\text{with fractions}&\text{simplified}\\
\hline
1&\dfrac{5\frac{0}{~}\cdot 3}{8\frac{1}{~}}&\dfrac{3}{8}&\dfrac{3}{8}\\\hline
2&\dfrac{5\frac{1}{~}\cdot 3}{8\frac{2}{~}}&\dfrac{5}{8}\cdot\dfrac{3}{7}&\dfrac{15}{56}\\\hline
3&\dfrac{5\frac{2}{~}\cdot 3}{8\frac{3}{~}}&\dfrac{5}{8}\cdot\dfrac{4}{7}\cdot\dfrac{3}{6}&\dfrac{5}{28}\\\hline
4&\dfrac{5\frac{3}{~}\cdot 3}{8\frac{4}{~}}&\dfrac{5}{8}\cdot\dfrac{4}{7}\cdot\dfrac{3}{6}\cdot\dfrac{3}{5}&\dfrac{3}{28}\\\hline
5&\dfrac{5\frac{4}{~}\cdot 3}{8\frac{5}{~}}&\dfrac{5}{8}\cdot\dfrac{4}{7}\cdot\dfrac{3}{6}\cdot\dfrac{2}{5}\cdot\dfrac{3}{4}&\dfrac{3}{56}\\
\hline6&\dfrac{5\frac{5}{~}\cdot 3}{8\frac{6}{~}}&\dfrac{5}{8}\cdot\dfrac{4}{7}\cdot\dfrac{3}{6}\cdot\dfrac{2}{5}\cdot\dfrac{1}{4}\cdot\dfrac{3}{3}&\dfrac{1}{56}\\\hline\end{array}$
As a sanity check, the values in the right column all add up to equal $1$, as expected.
Continue.

As an aside, yet another way you could arrive at the correct probability distribution is through the use of binomial coefficients.  For the $k$'th ball to be the first white one, we need to select $k-1$ red balls and one white ball and further need the white ball selected to occur last.
This occurs with probability $Pr(X=k)=\dfrac{\binom{5}{k-1}\binom{3}{1}}{\binom{8}{k}}\cdot\frac{1}{k}$ which one can check equals the same expression as discussed earlier.

It seems you are insistent on writing this with factorials.  The expression will be
$$Pr(X=k) = \dfrac{ \frac{5!}{(5-k+1)!}\cdot 3}{\frac{8!}{(8-k)!}}$$
Which doesn't simplify much.  If you insisted you could move things around as
$$Pr(X=k)= \dfrac{ (8-k)!}{112\cdot (5-k+1)!}$$
