Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

43
votes
3answers
8k views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
26
votes
4answers
12k views

Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them. According to Wikipeda: A Markov chain is a memoryless, random process. A ...
22
votes
4answers
24k views

Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I want ...
13
votes
1answer
711 views

Hilbert's Barber Shop

Hilbert opens a barber shop with an infinite number of chairs and an infinite number of barbers. Customers arrive via a Poisson random process with an expected 1 person every 10 minutes. Upon arrival, ...
13
votes
2answers
469 views

Difference in probability distributions from two different kernels

I wonder if the probability kernels of Markov processes on the same state space are close enough, does it also hold for the probabilities of the event that depend only on first $n$ values of the ...
12
votes
3answers
307 views

Select a new value from last $N$ values; how long until the last $N$ are all the same?

Say first we have N distinct numbers in a line, like 1,2,3,...,N, in each round, we choose a ...
12
votes
1answer
1k views

Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?

Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels. Feynman-Kac formula is also a pde corresponding to a stochastic process ...
10
votes
1answer
189 views

Exploiting the Markov property

I've encountered the following problem when dealing with short-rate models in finance and applying the Feynman-Kac theorem to relate conditional expectations to PDEs. Let $(\Omega,\mathcal{F},\{\...
10
votes
0answers
311 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. $Q=(q_{i,j})_{i,j\...
9
votes
1answer
962 views

Show that Brownian motion on the unit circle is exponentially ergodic and has the uniform measure as its invariant distribution.

My search results keep bring up planar Brownian motion on the unit disk. However, I am specifically referring to $e^{jW_{t}} = [\cos(W_t),\sin(W_t)]^{T}$ where $W_t$ is Brownian motion. I am at a ...
9
votes
0answers
276 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
9
votes
1answer
166 views

Mean hitting time: reference request

After answering [this question] (Expectation of a stopping time uniquely determined by a function) I was looking for the literature on the mean hitting/exit time for a discrete-time Markov process. ...
8
votes
2answers
3k views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
8
votes
1answer
4k views

Markov chains: is “aperiodic + irreducible” equivalent to “regular”?

I have two books on stochastic processes. In one book, it says that the limiting matrix is possible to find if the matrix is regular, that is, if for some $n$ $P^n$ has only positive values. The ...
8
votes
1answer
3k views

Why does a time-homogeneous Markov process possess the Markov property?

Klenke defines (Definition 17.3, p. 346) a time-homogeneous Markov process independently, rather than as a special case of a stochastic process that possesses the Markov property (Definition 17.1, p. ...
8
votes
1answer
13k views

Kendall notation's “General distribution”, what does that mean?

The first and second parameters for the Kendall's notation may have a G value, which stands for General distribution, see here. But what does that mean? What is a ...
8
votes
3answers
145 views

Disease spread through checkerboard

Suppose we have an infinite checkerboard (square grid) with a single “infected” square at time $t=0$. After each discrete time step, each square that is adjacent (sharing an edge) to one or more ...
7
votes
4answers
1k views

Probability of going into an absorbing state

If I have a random walk Markov chain whose transition probability matrix is given by $$ \mathbf{P} = \matrix{~ & 0 & 1 & 2 & 3 \\ 0 & 1 & 0 & 0 & 0 \\ ...
7
votes
1answer
6k views

Transition Kernel of a Markov Chain

Supposing $X_t$ is a Markov Process, can the transition kernel be defined by $$K_t(x,A):= P(X_{t+1} \in A | X_t = x)?$$ Assume that $X_t : \Omega \to \mathbb{R}^n$. The issue is that under the normal ...
7
votes
1answer
153 views

If $A$ is the generator of $(P_t)$, then $A+f$ is the generator of $(P_t^f)$

Let $X=(X_t)_{t\geq0}$ be a Markov process on a state space $\Gamma$ (a Hausdorff topological vector space), let $A$ be the infinitesimal generator of $X$ and let $\mathcal C(\Gamma)$ the space of ...
7
votes
1answer
227 views

If $X^{(n)},X$ are càdlàg and $X^{(n)}\to X$ in distribution, do the corresponding transition semigroups strongly converge?

Let $\left(\kappa^{(n)}_t\right)_{t\ge0}$ and $(\kappa_t)_{t\ge0}$ be Markov semigroups on $(\mathbb R,\mathcal B(\mathbb R))$ for $n\in\mathbb N$ $(T_n(t))_{t\ge0}$ and $(T(t))_{t\ge0}$ be strongly ...
7
votes
4answers
318 views

Discover where Bob is sleeping using hidden Markov chains

Bob lives in four different houses $A, B, C$ and $D$ that are connected like the following graph shows: Bob likes to sleep in any of his houses, but they are far apart so he only sleeps in a house ...
6
votes
5answers
2k views

Have any discrete-time continuous-state Markov processes been studied?

I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state ...
6
votes
1answer
511 views

How is the Chapman-Kolmogorov Equation not a fancy name for matrix multiplication?

The Chapman-Kolmogorov Equation: $$p^{m+n}(i,j)=\sum_kp^m(i,k)p^n(k,j)$$ Matrix Multiplication (with $[A]_{i,j}=a_{i,j}$ where $A$ is a linear map "" for B) $$[AB]_{i,j}=\sum_ka_{i,k}b_{k,j}$$ In ...
6
votes
3answers
582 views

Does an n-order Markov chain still represent a Markov process?

I am trying to understand Markov processes but am still confused by their definition. In particular, the Wikipedia page gives this example of a non-Markov process. The example is of pulling different ...
6
votes
1answer
151 views

Continuous-time Markov Chain question

Consider an immigration-death model $X = (X_t)_{t\geq0}$, i.e. a model where immigrants arrive according to a Poisson process with rate $\lambda$ and individuals have independent $Exp(\mu)$ lifetimes. ...
6
votes
1answer
281 views

Question about Markov chain

We know that if $\{X_n\}$ is a Markov chain, then $X_{n+1}$ is independent with the past states $X_0,\ldots,X_{n-1}$ given current state $X_n$, that is $$P\{X_{n+1}=j|X_0=i_0,\ldots,X_n=i\}=P\{X_{n+1}=...
6
votes
1answer
560 views

Infinitesimal generator of Brownian motion with additional jumps

A compound Poisson process is a jump process with two parameters, the rate of the jumps $\lambda$ and the distribution of the jumps $\mu$ ($\mu$ is a probability measure on $\mathbb{R}$). The ...
6
votes
1answer
1k views

Multidimensional infinitesimal generator of a jump-diffusion

Let $X=\{X_t\}_{t\geq0}$ be an $n$-dimensional Markov process, defined by the SDE $$dX_t = \mu(t, X_t) \, dt + \sigma(t,X_t) \, dB_t+\beta(t-,X_{t-}) \, dN_t,$$ where $\mu, \sigma$ and $\beta$ are ...
6
votes
1answer
198 views

Convergence of the distribution of the Langevin diffusion to its invariant measure

Let $(X_t)_{t\ge0}$ be a solution of $${\rm d}X_t=-h'(X_t){\rm d}t+\sqrt 2W_t,\tag1$$ where $(W_t)_{t\ge0}$ is a Brownian motion and $h$ is such that $X$ is the unique strong solution of $(1)$. Assume ...
6
votes
1answer
189 views

If $M$ maps all probability vectors on a subspace to some probability vectors, is $M$ the restriction of a column stochastic matrix?

For $n \ge 1$, let $$\Delta^{n-1} := \left\{ (x_1,\dots,x_{n}) \in \mathbb{R}^{n} \mid \sum_{i=1}^{n} x_i=1,~x_i \ge 0 \right\}$$ and let $\mathcal{S},~\mathcal{S}'\subset\mathbb{R}^n$ be subspaces ...
6
votes
1answer
138 views

Martingale converges to the boundary

I asked an almost same question before and it is solved by considering adjacent $Z_n$ can not be far away and obtain a contradiction. However, if the setting is altered a bit, I wonder whether it is ...
6
votes
0answers
251 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
6
votes
1answer
432 views

markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, ...
5
votes
2answers
5k views

Transformation of state-space that preserves Markov property

I am solving a problem in Mathematical Statistics by Jun Shao Let $\{X_n \}$ be a Markov chain. Show that if $g$ is a one-to-one Borel function, then $\{g(X_n )\}$ is also a Markov chain. Give an ...
5
votes
1answer
2k views

Ornstein-Uhlenbeck process: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not a ...
5
votes
2answers
11k views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} &...
5
votes
2answers
2k views

Diffusion process. Distribution vs transition probability.

I need confirmation on the following problem: Take a SDE of the form: \begin{equation} dX_t=a(X_t,t)dt+b(X_t,t)dW_t \end{equation} where all the conditions, such that the solution $X_t$ is defined ...
5
votes
2answers
1k views

Is every Markov Process a Martingale Process?

According to the definition (2.3.6) of a Markov Process in Shreve's book titled Stochastic Calculus for Finance II: Let $(\Omega,\mathcal F,\mathbb P)$ be a probability space, let $T$ be a fixed ...
5
votes
2answers
2k views

Convergence of the powers of a Markov transition matrix

I have a Markov matrix $$P=\begin{bmatrix}1&0&0&0&0&0\\\frac{1}{2}&0&\frac{1}{2}&0&0&0\\\frac{1}{4}&0&\frac{1}{4}&\frac{1}{2}&0&0\\\frac{1}{...
5
votes
1answer
4k views

Find the Stationary Distribution of an infinite state Markov chain

A Markov Chain on states 0,1,..... has transition probabilities $P_{ij}=1/(i+2)$ for j=0,1,....,i,i+1. I'm supposed to find the stationary distribution. So do I take the limit as n goes to ...
5
votes
1answer
171 views

Markov chains. If $X_0=0$ then the probability that $X_n\ge1$ for all $n\ge 1$ is $\frac{6}{\pi^2}$

Let $(X_n)_{n\ge0}$ be a markov chain on $\{0,1,...\}$ with transition probabilities given by $p_{01}=1,p_{i,i+1}+p_{i,i-1}=1, p_{i,i+1}=\Big(\frac{i+1}{i}\Big)^2p_{i,i-1}, i\ge1$ I need to show ...
5
votes
1answer
288 views

Constructing Martingales from Markov Processes

I know that for a Markov process $X_t$ with generator $L$ and $f,f^2\in D(L)$, $$M_t=f(X_t)-\int_0^t Lf(X_s)\ ds$$ is a martingale (w.r.t. $P^x$). And I want to show that $$M_t^2-\int_0^t (Lf^2(X_s)-...
5
votes
1answer
986 views

Covariance of Brownian-motion-like processes

We know that $\operatorname{Cov}(B_s,B_t)=\min(s,t)$ if $B_t$ is Brownian motion. What is $\operatorname{Cov}(B_{f(s)},B_{f(t)})$ for some injective $f$? How can I write $B_{f(t)}$ in an Ito ...
5
votes
1answer
535 views

Convergence of discrete-time Markov chain to Feller processes

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(X_t)_{t\ge0}$ be a Feller process on $(\Omega,\mathcal A,\operatorname P)$ $(h_d)_{d\in\mathbb N}\subseteq(0,\infty)$ with $$h_d\...
5
votes
1answer
1k views

Joint distribution of Brownian motion and its running maximum

$B$ being standard Brownian motion, its running maximum is defined as $M_t = \sup_{0\leq s\leq t} B_s$. I am trying to follow the proof of the following result but I don't understand some of the steps ...
5
votes
1answer
147 views

Log-likelihood function

I'm not sure if this could be asked here, or in math overflow... In the following paper Cho, Jin Seo, and Halbert White. "Testing for regime switching." Econometrica 75.6 (2007): 1671-1720. doi: 10....
5
votes
1answer
164 views

Entropy of bits on a chessboard

Suppose we have an $n$ by $n$ chessboard and each square can be in two states: "on" or "off". At time $t=0$ all bits are off. Every second one square is selected at random uniformly out of the $n^2$ ...
5
votes
1answer
173 views

A proof by René Schilling that a continuous Lévy process is integrable

In his treatise "An Introduction to Lévy and Feller Processes" (arXiv link), Prof. Dr. René Schilling gives a short and seemingly straightforward proof for the claim that a continuous Lévy process is ...
5
votes
1answer
3k views

Ehrenfest Chain: stationary distribution

In the Ehrenfest Chain model: There are M balls which are divided between urn A and urn B. At each stage, if a ball is chosen, then it would be moved into a different urn. Let $X_n$ be the # of ...