Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

Question

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them.

According to Wikipeda:

• A Markov chain is a memoryless, random process.

• A Markov process is a stochastic process, which exhibits the Markov property.

• The Markov property is the memorylessness of a stochastic property.

• A stochastic process is a random process, which is a collection of random variables.

• And finally, random variables are those determined by chance instead of other variables, which seems to mean explicitly that they are memoryless.

Thus, it seems that stochastic process, random process, Markov chain, and Markov process are all the exact same thing... which is a collection of random variables, which are memory-less, which means they exhibit the Markov property.

Answer 1

The difference between Markov chains and Markov processes is in the index set, chains have a discrete time, processes have (usually) continuous.

Random variables are much like Guinea pigs, neither a pig, nor from Guinea. Random variables are functions (which are deterministic by definition). They are defined on probability space which most often denotes all possible outcomes of your experiment/model. In schools their value set is almost always a subset of $\mathbb{R}$.

Sequences of random variables don't need to be memoryless, e.g. sequences of random variables that denote some cumulative usually aren't memoryless. On the other hand, for example, sequences of independent identically distributed random variables do not depend on time at all, and so they have to be memoryless. Those two examples are something like extremes, where the next variable in the sequence depends on all of the previous (in the former example), or none of them (in the latter). The Markov property tells us, they may depend, but if they do, it does not give us any more information (e.g. in the case of discrete time, that is, Markov chains, it means that the next can be determined only using the current and nothing else). Finally, note that there is a difference between "does not depend" and "does not give us any new information", for example consider a Markov chain defined on the set of finite binary sequences where each step adds a (uniformly) random bit. Clearly, the next state does depend on all the previous "coin flips" (these are embedded in the bits of sequence prefix), but the current state already contains everything we need.

I hope it explained something $\ddot\smile$

Answer 2

Although there is no definitive agreement in the literature on the use of Markov process and Markov chain, most authors distinguish both terms based on whether the state space is continuous or discrete. When the state space is continuous, it is called a Markov process regardless of whether the parameter (or time) is discrete or continuous. When the Markov process is discrete-valued (i.e discrete state space), it is called a Markov chain. Markov chain can further be divided into Continuous –parameter (or time) Markov chain and discrete – parameter (or time) Markov chain for continuous and discrete parameters respectively.

Random process and stochastic process are synonymous. Both map elements of the sample space into functions of time.

Random variable on the other hand is a function that maps elements of the sample space into points of the real axis.

References: (1) Signal detection and estimation by Mourad Barkat, 2nd edition (2) Schaum's outlines (Probability, Random Variables, and Random Processes)

Answer 3

A random variable is just a mathematical object designed to capture a notion of uncertainty about the value that the variable takes. A simple example of a random variable is the outcome of a (not necessarily fair) coin toss. We know the outcome must be one of $H$ or $T$, but we don't know which until we've actually tossed the coin, allowing the uncertainty to resolve itself.

Stochastic is, in the words of my stochastic analysis professor, a posh term for 'random'. Hence, stochastic processes and random processes are essentially the same thing.

A stochastic process is just a special type of random variable. You can think of it as a collection of random variables, with some index -- typically time. Continuing with the coin tossing example, a (finite or infinite) sequence of coin tosses is a stochastic process. Typical in the theory of stochastic processes is that the index set, which we'll think of as time, has some special properties. For instance, the index might be related to the revelation of information. In the coin tossing example, you might naturally assume that you only observe each coin toss one at a time, and when you know the outcome of the $n^{\text{th}}$ coin toss, you also know the outcomes of preceding coin tosses.

Markov chains/processes (I, like Wikipedia, prefer to think of them as the same thing, but others might want to make some technical distinctions between them) are a special type of stochastic process. In particular, their distinguishing feature is the Markov property of memorylessness.

However, not all stochastic processes are memoryless. A simple (albeit admittedly not very natural) example might be a coin whose probability of coming up $H$ depends on the outcome of last $5$ coin tosses (whenever you've tossed it at least $5$ times). On the other hand, a canonical example of a Markov process is a random walk.

One way to think about the Markov property is to ask yourself if more observations of past outcomes other than the most recent one gives you more information about the likely future outcomes of the stochastic process. If observations other than the most recent one contain no additional information, the process is Markov. My strange example is clearly not Markov, since you would have different beliefs over the distribution of the next toss depending on whether you knew only the most recent outcome, or outcomes before that. The random walk is Markov since where the path currently is tells you everything about where it is going to go next -- additional observations of its previous history gives you no additional information.

A more natural example of a stochastic process that isn't Markov appears in the Wikipedia article on Markov Chains. This is essentially the same as drawing differently coloured balls from an urn without replacement, which is a common source of examples in probability theory.

Answer 4

The Wikipedia article on stochastic processes has been recently re-written to clarify this issue, among others. (Unfortunately the articles on specific stochastic processes such as Markov chains still need a lot of work.)

https://en.wikipedia.org/wiki/Stochastic_process#Markov_processes_and_chains

Usually a Markov chain is a specific type of Markov process. Unfortunately, as some have already mentioned, there is no agreed upon definition for a Markov process, except that either the state space or index set has to be discrete. For example, when people talk about a continuous-time Markov chain, they are using the convention that Markov chain refers to a Markov process with discrete state space. I would guess this is the more popular convention, but many use Markov chain to refer to Markov process with discrete time. I suppose, logically, if you accept both conventions then the terms "Markov process" and "Markov chain" become completely interchangeable.

The term Markov chain is older, being used (in French) at least in the 1920s. Andrey Kolmogovov did pioneering in the 1930s work on Markov processes with continuous state space, but he didn't use the term "Markov process". William Feller decided to use that term in the 1940s.

Markov processes and, consequently, Markov chains are both examples of stochastic processes.

Random process and stochastic process are completely interchangeable (at least in many books on the subject). See relevant citations:

https://en.wikipedia.org/wiki/Stochastic_process#Terminology

Stochastic processes are, in general, not memoryless, but the term "memoryless" needs a precise definition. One could have, for example, a discrete-time stochastic process that depends on its states for the 3 previous times, but not all times, so in the long run it forgets all. Other processes only depend on the current state.