From Earliest Known Uses of Some Words in Mathematics, one can read the entry:
ERGODIC. Ludwig Boltzmann (1844-1906) coined the term Ergode (from the Greek words for work + way) for what Gibbs later called a "micro-canonical ensemble"; Ergode appears in the 1884 article in Wien. Ber. 90, 231. Later P. & T. Ehrenfest (1911) "Begriffiche Grundlagen der statistischen Auffassung in der Mechanik" (Encyklopädie der mathematischen Wissenschaften, vol. 4, Part 32) discussed "ergodische mechanischer Systeme" the existence of which they saw as underlying the gas theory of Boltzmann and Maxwell. (Based on a note on p. 297 of Lectures on Gas Theory, S. G. Brush's translation of Boltzmann's Vorlesungen über Gastheorie.)
After the impossibility of an ergodic mechanical system was demonstrated, various related hypotheses were investigated. "Ergodic" and "quasi-ergodic" theorems were proved in the 1930s, by, amongst others, G. D. Birkhoff in Proc. Nat. Acad. Sci. (1931) 17, 651 -- "I propose ... to establish a general recurrence theorem and thence the 'ergodic theorem'" -- and J. von Neumann Proc. Nat. Acad. Sci. (1932) 18, 70-82.
Ergodic theorems originated in classical mechanics but in the theory of stochastic processes they appear as versions of the law of large numbers, see e.g. J. L. Doob's Stochastic Processes (1954). [This entry was contributed by John Aldrich.]
From mathworld's entry on Ergodic Theory:
Ergodic theory had its origins in the work of Boltzmann in statistical mechanics problems where time- and space-distribution averages are equal. Steinhaus (1999, pp. 237-239) gives a practical application to ergodic theory to keeping one's feet dry ("in most cases," "stormy weather excepted") when walking along a shoreline without having to constantly turn one's head to anticipate incoming waves. The mathematical origins of ergodic theory are due to von Neumann, Birkhoff, and Koopman in the 1930s. It has since grown to be a huge subject and has applications not only to statistical mechanics, but also to number theory, differential geometry, functional analysis, etc. There are also many internal problems (e.g., ergodic theory being applied to ergodic theory) which are interesting.
References cited there:
Billingsley, P. Ergodic Theory and Information. New York: Wiley, 1965.
Cornfeld, I.; Fomin, S.; and Sinai, Ya. G. Ergodic Theory. New York: Springer-Verlag, 1982.
Katok, A. and Hasselblatt, B. An Introduction to the Modern Theory of Dynamical Systems. Cambridge, England: Cambridge University Press, 1996.
Nadkarni, M. G. Basic Ergodic Theory. India: Hindustan Book Agency, 1995.
Parry, W. Topics in Ergodic Theory. Cambridge, England: Cambridge University Press, 1982.
Petersen, K. Ergodic Theory. Cambridge, England: Cambridge University Press, 1983.
Radin, C. "Ergodic Theory." Ch. 1 in Miles of Tiles. Providence, RI: Amer. Math. Soc., pp. 17-54, 1999.
Sinai, Ya. G. Topics in Ergodic Theory. Princeton, NJ: Princeton University Press, 1993.
Smorodinsky, M. Ergodic Theory, Entropy. Berlin: Springer-Verlag, 1971.
Steinhaus, H. Mathematical Snapshots, 3rd ed. New York: Dover, pp. 237-239, 1999.
Walters, P. Ergodic Theory: Introductory Lectures. New York: Springer-Verlag, 1975.
Walters, P. Introduction to Ergodic Theory. New York: Springer-Verlag, 2000.
See also Wikipedia's entry on Ergodic Theory, which discusses its current usage and applications, and includes both historic and modern references might be of interest.