If we have infinite variance, can the expectation 'mean' anything? If we have a random variable $X$ with infinite variance $Var(X)$ then how can the expectation $E(X)$ be useful to us? In the sense that our values vary so much from it that it holds no relevance. Maybe an example would clear things up?
 A: Maybe this example will help clear things up:
Suppose I am taking very expensive samples from two normal
populations with the same unknown population variance, and I want to
test whether the two population means are equal. I can only afford
to take two observations from each population. 
Maybe these are observations from space rocket launches or 
from some rare tran-uranic element made only in atom smashers. 
So this will be
a pooled two-sample t test and (under the null hypothesis of equal
populations means) the t statistic will have Student's t distribution with
$n_1 + n_2 - 2 = 2$ degrees of freedom. This distribution has expectation 0
and infinite (or non-existent) variance (depending on definitions).
My first sample gives values 1 and 2; the second gives 101 and 102.
The pooled t test procedure in R gives output as follows:
t.test(x1, x2, var.eq=2)

        Two Sample t-test

data:  x1 and x2
t = -141.42, df = 2, p-value = 5e-05
alternative hypothesis: true difference in means is not equal to 0
sample estimates:
mean of x mean of y 
      1.5     101.5 

So the P-value is minuscule. I can pretty confidently claim that the
population means are not equal. Student's t distribution
(with a mean and infinite variance) has given me a useful result.
Even though the distribution has infinite variance I can find
probabilities in the tail of the distribution that are meaningful.
Note: Although this is a standard frequentist test of hypothesis, and
frequentist statisticians enjoy believing that they do not 'drag prior
information' into their inferences, one has to admit that this situation is
heavily laden with 'information' beyond the data: (1) The information
that the population is normally distributed and (2) the assumption
that the two populations have equal variances are both superimposed
on the structure of the test before we see any data. However, I think
the example holds its own in showing that distributions without variances
can be useful in statistical inference.
A: Some basic yet useful properties of expectation hold without any assumptions on variance. For example, if $X$ is a nonnegative random variable, then
$$
   \Pr[X \ge t] \le \frac{\mathbb E[X]}{t}
$$
is a useful bound on the tail probability of a heavy-tailed distribution.
