# Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them.

According to Wikipeda:

• A Markov chain is a memoryless, random process.

• A Markov process is a stochastic process, which exhibits the Markov property.

• The Markov property is the memorylessness of a stochastic property.

• A stochastic process is a random process, which is a collection of random variables.

• And finally, random variables are those determined by chance instead of other variables, which seems to mean explicitly that they are memoryless.

Thus, it seems that stochastic process, random process, Markov chain, and Markov process are all the exact same thing... which is a collection of random variables, which are memory-less, which means they exhibit the Markov property.

• I had the same problem with the Wikipedia pages on Markov stuff. ARGH. – Doctor Mohawk Dec 11 '14 at 22:00
• This is the funniest question I have seen so far in math.exchange! – nbro Dec 15 '15 at 15:58

The difference between Markov chains and Markov processes is in the index set, chains have a discrete time, processes have (usually) continuous.

Random variables are much like Guinea pigs, neither a pig, nor from Guinea. Random variables are functions (which are deterministic by definition). They are defined on probability space which most often denotes all possible outcomes of your experiment/model. In schools their value set is almost always a subset of $\mathbb{R}$.

Sequences of random variables don't need to be memoryless, e.g. sequences of random variables that denote some cumulative usually aren't memoryless. On the other hand, for example, sequences of independent identically distributed random variables do not depend on time at all, and so they have to be memoryless. Those two examples are something like extremes, where the next variable in the sequence depends on all of the previous (in the former example), or none of them (in the latter). The Markov property tells us, they may depend, but if they do, it does not give us any more information (e.g. in the case of discrete time, that is, Markov chains, it means that the next can be determined only using the current and nothing else). Finally, note that there is a difference between "does not depend" and "does not give us any new information", for example consider a Markov chain defined on the set of finite binary sequences where each step adds a (uniformly) random bit. Clearly, the next state does depend on all the previous "coin flips" (these are embedded in the bits of sequence prefix), but the current state already contains everything we need.

I hope it explained something $\ddot\smile$

• What about Markov models? How do they tie in with chains and processes? – Doctor Mohawk Dec 11 '14 at 22:02
• @mohawkjohn Markov modeling is the general term for any modelling technique that uses Markov chains or processes. – dtldarek Dec 11 '14 at 23:06
• So a Continuous Time Markov Chain (CTMC) is just an other name for Markov Process? – Bakuriu Jul 20 '16 at 13:40
• @Bakuriu I would say Continuous Time Markov Process instead of CTMC, but that's personal preference. Markov Process is a general name for a stochastic process with the Markov Property – the time might be discrete or not. Because most authors use term "chain" in the discrete case, then if somebody uses term "process" the usual connotation is that we are looking at non-discrete time, but in fact we don't know, it could be either way. The term "Countinuous Time Markov Chain/Process" explicitly states the time is continuous, so it is more specific than just "Markov Process". – dtldarek Jul 20 '16 at 15:58
• "(in Markov chains) the next depends only on the current and nothing else" That is misleading at best, wrong at worst. A (first order) Markov chain has long-term dependencies. What it's true is that, given the current value the past values don't give more information (or: the past conditioned on the present is independent of the future) – leonbloy Dec 16 '16 at 3:59

Although there is no definitive agreement in the literature on the use of Markov process and Markov chain, most authors distinguish both terms based on whether the state space is continuous or discrete. When the state space is continuous, it is called a Markov process regardless of whether the parameter (or time) is discrete or continuous. When the Markov process is discrete-valued (i.e discrete state space), it is called a Markov chain. Markov chain can further be divided into Continuous –parameter (or time) Markov chain and discrete – parameter (or time) Markov chain for continuous and discrete parameters respectively.

Random process and stochastic process are synonymous. Both map elements of the sample space into functions of time.

Random variable on the other hand is a function that maps elements of the sample space into points of the real axis.

References: (1) Signal detection and estimation by Mourad Barkat, 2nd edition (2) Schaum's outlines (Probability, Random Variables, and Random Processes)

• Disagree on random variables. The values of a random variable may not be real, and may not be number. Otherwise, if you threw a coin and looked at its outcome, if you encoded the outcome with $\{0,1\}$ (as real numbers), you would get a random variable, while if you encoded it with $\{H, T\}$, you wouldn't. Which is very inelegant. – D. Thomine Jan 6 '17 at 13:22
• D. Thomine, I think you are mixing up sample point and the values a rv can take. rv is real-valued. Using the example you gave, both {0.1} and {H,T} are ways of encoding the outcomes of an experiment. Hence, they are elements of the sample space (i.e sample point). A rv is a function that will take these sample points as argument and assign real values to them, e.g a rv may assign 0 to the outcome "head" (which may have been encoded as 0 or H) and 1 to the outcome "tail" (encoded as 1 or T). Thus, the sample space is the domain of the rv while the collection of all numbers is its range. – Henry Ogworonjo Jan 8 '17 at 1:15

A random variable is just a mathematical object designed to capture a notion of uncertainty about the value that the variable takes. A simple example of a random variable is the outcome of a (not necessarily fair) coin toss. We know the outcome must be one of $H$ or $T$, but we don't know which until we've actually tossed the coin, allowing the uncertainty to resolve itself.

Stochastic is, in the words of my stochastic analysis professor, a posh term for 'random'. Hence, stochastic processes and random processes are essentially the same thing.

A stochastic process is just a special type of random variable. You can think of it as a collection of random variables, with some index -- typically time. Continuing with the coin tossing example, a (finite or infinite) sequence of coin tosses is a stochastic process. Typical in the theory of stochastic processes is that the index set, which we'll think of as time, has some special properties. For instance, the index might be related to the revelation of information. In the coin tossing example, you might naturally assume that you only observe each coin toss one at a time, and when you know the outcome of the $n^{\text{th}}$ coin toss, you also know the outcomes of preceding coin tosses.

Markov chains/processes (I, like Wikipedia, prefer to think of them as the same thing, but others might want to make some technical distinctions between them) are a special type of stochastic process. In particular, their distinguishing feature is the Markov property of memorylessness.

However, not all stochastic processes are memoryless. A simple (albeit admittedly not very natural) example might be a coin whose probability of coming up $H$ depends on the outcome of last $5$ coin tosses (whenever you've tossed it at least $5$ times). On the other hand, a canonical example of a Markov process is a random walk.

One way to think about the Markov property is to ask yourself if more observations of past outcomes other than the most recent one gives you more information about the likely future outcomes of the stochastic process. If observations other than the most recent one contain no additional information, the process is Markov. My strange example is clearly not Markov, since you would have different beliefs over the distribution of the next toss depending on whether you knew only the most recent outcome, or outcomes before that. The random walk is Markov since where the path currently is tells you everything about where it is going to go next -- additional observations of its previous history gives you no additional information.

A more natural example of a stochastic process that isn't Markov appears in the Wikipedia article on Markov Chains. This is essentially the same as drawing differently coloured balls from an urn without replacement, which is a common source of examples in probability theory.

The Wikipedia article on stochastic processes has been recently re-written to clarify this issue, among others. (Unfortunately the articles on specific stochastic processes such as Markov chains still need a lot of work.)

https://en.wikipedia.org/wiki/Stochastic_process#Markov_processes_and_chains

Usually a Markov chain is a specific type of Markov process. Unfortunately, as some have already mentioned, there is no agreed upon definition for a Markov process, except that either the state space or index set has to be discrete. For example, when people talk about a continuous-time Markov chain, they are using the convention that Markov chain refers to a Markov process with discrete state space. I would guess this is the more popular convention, but many use Markov chain to refer to Markov process with discrete time. I suppose, logically, if you accept both conventions then the terms "Markov process" and "Markov chain" become completely interchangeable.

The term Markov chain is older, being used (in French) at least in the 1920s. Andrey Kolmogovov did pioneering in the 1930s work on Markov processes with continuous state space, but he didn't use the term "Markov process". William Feller decided to use that term in the 1940s.

Markov processes and, consequently, Markov chains are both examples of stochastic processes.

Random process and stochastic process are completely interchangeable (at least in many books on the subject). See relevant citations:

https://en.wikipedia.org/wiki/Stochastic_process#Terminology

Stochastic processes are, in general, not memoryless, but the term "memoryless" needs a precise definition. One could have, for example, a discrete-time stochastic process that depends on its states for the 3 previous times, but not all times, so in the long run it forgets all. Other processes only depend on the current state.