# Have any discrete-time continuous-state Markov processes been studied?

I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state Markov processes (such as Brownian motions).

I was wondering if discrete-time continuous-state Markov processes have been studied as often as the above three?

What are some of its examples then?

• Is the discrete time Brownian motion an example? – user60610 Apr 14 '13 at 22:45
• What is "the discrete time Brownian motion"? @TongZhang – Tim Apr 14 '13 at 22:49
• I guess this is not so widely studied. In my stochastic calculus class the notes talked about discrete time BM before the introduction of BM. – user60610 Apr 14 '13 at 22:53
• @TongZhang: thanks, can you link me to your notes or some references for that? – Tim Apr 14 '13 at 23:00
• Here is the course web in which I learned this thing. I did not realize that it was so wired until I scanned nearly all the first 100 pages shown in google search and found nothing about it. The notes is unfortunately not available online. Now I start to think maybe it is not of value at all but only for some introductory purpose. – user60610 Apr 14 '13 at 23:00

Discrete-time continuous state Markov processes are widely used. Autoregressive processes are a very important example.

Actually, if you relax the Markov property and look at discrete-time continuous state stochastic processes in general, then this is the topic of study of a huge part of Time series analysis and signal processing.

The most famous examples are ARMA processes, the Conditionally Heteroscedastic models, a large subclass of Hidden Markov models....

Some inventory models which assume the products are continuous are actually continuous state markov process. In operations research, people seems to care more about the structural results instead of the computation issue when they assume the state is continuous. This maybe the reason that it is not well known. Some reference of Continuous state Markov chain are: (1) S P Meyn and R L Tweedie. Markov Chains and Stochastic Stability. Cambridge University Press, 2009 (2) QUANTITATIVE ECONOMICS with Python

The references I provide may not contain many examples you are after, but they do treat theory that may help you understand examples you find elsewhere.

The book titled Probability by A.N. Shiryaev, with ISBN 978-1-4757-2541-4 (e-book ISBN 978-1-4757-2539-1) has a chapter called CHAPTER VIII; Sequences of Random Variables that Form Markov Chains.

This chapter has strong foundations in measure theoretic probability theory. That subject is dealt with in earlier chapters of the book.

It should be noted that (unless I overlooked any) the examples in this chapter deal only with discrete state Markov processes.

Note

I referred to the second edition of this book. There is a newer version of this book with ISBN 978-0-387-72206-1, which is the third edition. I refer to the second edition because I cannot find a table of contents (etc) for the third edition.

Another reference

The book The Theory of Stochastic Processes I, isbn: 978-3-642-61943-4, which can be found here has a chapter called Random Sequences.

According to the book Probability this chapter deals with Markov Processes

The existence of regular transition probabilities such that the Kolmogorov-Chapman equation is satisfied for all $x \in \mathbb{R}$ is proved in [The Theory of Stochastic Processes] Volume I, Chapter II, §4).

The title of the chapter implies that it deals with discrete time processes and a look at the preview on springer reveals that the state (phase) space is taken to be general.

An easy example is as follows: let $(X_n)$ be an iid sequence of standard Gaussian random variables (although you don't usually think of it as a discrete-time / continuous-state Markov process).

I know that this is an old question, but I just stumbled upon it and noticed that nobody has mentioned yet what I think of as maybe the most natural and simple (non-trivial) examples of discrete-time Markov processes with non-discrete state space.

Starting with a continuous time Markov process $$X=(X_t)_{t\ge0}$$ on any state space $$E$$, one can consider

• the $$E$$-valued $$\Delta$$-grid chain $$(X_{n\Delta})_{n\in\Bbb N}$$ where $$\Delta>0$$,

• the $$E^{[0,\Delta)}$$-valued $$\Delta$$-path-segment chain $$((X_t)_{t\in[(n-1)\Delta,n\Delta)})_{n\in\Bbb N}$$ where $$\Delta>0$$,

• the $$E$$-valued sampled chain $$(X_{\tau_n})_{n\in\Bbb N}$$ where $$\tau_n=\sigma_1+\ldots+\sigma_n$$ with i.i.d. exponential random variables $$\sigma_i$$.

When looking for certain asymptotic properties of $$X$$, it can be sufficient (and yet much easier) to study one of these "embedded" discrete time processes. Such connections can be used even in the case where $$X$$ is not homogeneous but only $$\tau$$-periodic with respect to time, i.e. $$\mathcal L(X_{t+k\tau}|X_{s+k\tau})= \mathcal L(X_{t}|X_{s}) \quad\text{for all t\ge s\ge0 and k\in\Bbb N.}$$ In this situation, the advantage of the grid chain or the path-segment chain is even more obvious, since they become homogeneous when one takes $$\Delta=\tau$$.

To illustrate this a little more, think of an SDE $$dX_t=b(t,X_t)dt+\sigma(t,X_t)dW_t,$$ where drift and volatility are $$\tau$$-periodic in time. Many interesting properties of this continuous-time inhomogeneous Markov process can be derived from properties of the corresponding $$\tau$$-grid chain, which is a discrete-time homogeneous Markov process - and as such it is of a much simpler general nature and allows using the corresponding techniques.