Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

42
votes
3answers
8k views

What is the importance of the infinitesimal generator of Brownian motion?

I have read that the infinitesimal generator of Brownian motion is $\frac{1}{2}\small\triangle$. Unfortunately, I have no background in semigroup theory, and the expositions of semigroup theory I have ...
5
votes
2answers
5k views

Transformation of state-space that preserves Markov property

I am solving a problem in Mathematical Statistics by Jun Shao Let $\{X_n \}$ be a Markov chain. Show that if $g$ is a one-to-one Borel function, then $\{g(X_n )\}$ is also a Markov chain. Give an ...
5
votes
1answer
526 views

Convergence of discrete-time Markov chain to Feller processes

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(X_t)_{t\ge0}$ be a Feller process on $(\Omega,\mathcal A,\operatorname P)$ $(h_d)_{d\in\mathbb N}\subseteq(0,\infty)$ with $$h_d\...
2
votes
2answers
355 views

Is $(B_t^2)$ Markov where $(B_t)$ is Brownian motion?

I am pretty sure $(B_{t}^{2})$ not Markov because the squared random walk is not. Showing the square of a Markov process is or isn't Markov I guess I can repeat the method since to be Markov it ...
22
votes
4answers
24k views

Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I want ...
6
votes
1answer
191 views

Convergence of the distribution of the Langevin diffusion to its invariant measure

Let $(X_t)_{t\ge0}$ be a solution of $${\rm d}X_t=-h'(X_t){\rm d}t+\sqrt 2W_t,\tag1$$ where $(W_t)_{t\ge0}$ is a Brownian motion and $h$ is such that $X$ is the unique strong solution of $(1)$. Assume ...
3
votes
1answer
117 views

A variant of Kac's theorem for conditional expectations?

This is part of the proof of the Strong Markov property of Brownian motion given in Schilling's Brownian motion. Here $B_t$ is a $d$-dimensional Brownian motion with admissible filtration $\mathscr{...
1
vote
1answer
157 views

Symmetric random walk and the distribution of the visits of some state

I need help with this problem: Let $(S_n)_n$ a symmetric random walk in $\mathbb{Z}$ i.e $S_n=X_1 + \cdots + X_n$ with $(X_n)$ iid $\mathbb{P}(X_n=1)=\mathbb{P}(X_n=-1)=\frac{1}{2}$. Let $m \in \...
8
votes
1answer
3k views

Why does a time-homogeneous Markov process possess the Markov property?

Klenke defines (Definition 17.3, p. 346) a time-homogeneous Markov process independently, rather than as a special case of a stochastic process that possesses the Markov property (Definition 17.1, p. ...
5
votes
1answer
1k views

Joint distribution of Brownian motion and its running maximum

$B$ being standard Brownian motion, its running maximum is defined as $M_t = \sup_{0\leq s\leq t} B_s$. I am trying to follow the proof of the following result but I don't understand some of the steps ...
3
votes
1answer
506 views

Why is the following example of a Markov process not strong Markov

$X(t) := 0 \;\; (t \leq \tau),\;\; t - \tau\;\;(t \geq \tau)$ with $\tau$ exponentially distributed. Then X has the Markov property but not Strong Markov Property. But why ???? Can someone kindly ...
3
votes
1answer
235 views

A question about a stochastic process being Markov

Let $(X_{s},\mathcal{F}_{s})$ be a stochastic process adapted to a given filtration. I was told that, in order to prove that $X$ is Markov, it suffice to prove that for any nonnegative, Borel-...
0
votes
1answer
691 views

Flea on a triangle

"A flea hops randomly on the vertices of a triangle with vertices labeled 1,2 and 3, hopping to each of the other vertices with equal probability. If the flea starts at vertex 1, find the probability ...
4
votes
2answers
139 views

Under what conditions does $(I-N)^{-1}$ exist?

Given an nxn matrix N and $I=I_n$, under what conditions does $(I-N)^{-1}$ exist? On one hand $(I-N)(I + N + N^2 + ...) = (I + N + N^2 + ...) - (N + N^2 + ...) = I?$ On the other hand, $(I-N)(I + N +...
3
votes
1answer
73 views

An example of almost sure finite random time S such that $W_t=B_{S+t}-B_S$ is not a brownian motion

Let $\{B_t,\mathcal{F}_t,t \geq 0\}$ be a standard one dimensional Brownian motion. Give an example of a random time $S$ with $P[0\leq S< \infty]=1$ such that with $W_t:=B_{S+t}-B_S$, the process $...
1
vote
3answers
1k views

Markov Chain: flip 8 coins and get 3 consecutive heads

I was reading the material and I am confused at the following example. Q: In a sequence of independent flips of a fair coin, let N denote the number of flips until there is a run of three ...
1
vote
1answer
79 views

Is the transition semigroup of the solution of an SDE with Lipschitz coefficients strongly continuous on $C_b$?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $b,\sigma:\mathbb R\to\mathbb R$ be Lipschitz continuous (and hence at most of linear growth) and $$Lf:=bf'+\frac12\sigma^2f''\;\;\;\...
1
vote
0answers
131 views

Is the process Markov or not?

Consider the stochastic process with $X_0=0$ and $$ X_t= \begin{cases} 0 & \text{ for } \ \ t<\tau_1 \\ 1 & \text{ for } \ \ \tau_1\leq t < \tau_1+\tau_2 \\ 2 & \text{ for } \ \ \...
0
votes
1answer
870 views

Showing the square of a Markov process is or isn't Markov

Hi I am trying to show that if $X_n$ is a markov process, whether or not $X_n^2$ is a markov process. $X_n$ is a markov process if $P\{X_k = a_k|X_{k-1} = a_{k-1}, X_{k-2} = a_{k-2}, ..., X_k = a_1 \...
0
votes
1answer
174 views

probability, random walk, Markov chain question

Let $P$ be a transition matrix for a regular Markov chain and let $w$ be it’s equilibrium vector. Show that $w$ has no zero entries.
26
votes
4answers
12k views

Markov process vs. markov chain vs. random process vs. stochastic process vs. collection of random variables

I'm trying to understand each of the above terms, and I'm having a lot of trouble deciphering the difference between them. According to Wikipeda: A Markov chain is a memoryless, random process. A ...
12
votes
1answer
1k views

Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?

Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels. Feynman-Kac formula is also a pde corresponding to a stochastic process ...
9
votes
0answers
275 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
5
votes
1answer
2k views

Ornstein-Uhlenbeck process: Markov, but not martingale?

I'm puzzled about properties of the Ornstein-Uhlenbeck process, given by the Itō integral $$ X_t = x e^{-\lambda t} + \sigma \int_0^t e^{-\lambda(t-s)} d W_s \,. $$ I compute that $\{X_t\}$ is not a ...
4
votes
1answer
748 views

Prove that Brownian Motion absorbed at the origin is Markov

I'm trying to prove that Brownian motion absorbed at the origin is a Markov process with respect to the original filtration $\{\mathcal{F}_{t}\}$. To be more specific, let $(B_{t},\mathcal{F}_{t})_{t \...
5
votes
2answers
2k views

Convergence of the powers of a Markov transition matrix

I have a Markov matrix $$P=\begin{bmatrix}1&0&0&0&0&0\\\frac{1}{2}&0&\frac{1}{2}&0&0&0\\\frac{1}{4}&0&\frac{1}{4}&\frac{1}{2}&0&0\\\frac{1}{...
5
votes
2answers
11k views

Expected number of steps between states in a Markov Chain

Suppose I am given a state space $S=\{0,1,2,3\}$ with transition probability matrix $\mathbf{P}= \begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 \\[0.3em] \frac{2}{3} &...
4
votes
1answer
346 views

Is the infinitesimal generator for Lie groups the same as the infinitesimal generator of a Markov semigroup?

Is the infinitesimal generator for Lie groups related to the infinitesimal generator of a Markov semigroup? Or are they totally different concepts? https://en.wikipedia.org/wiki/Lie_group#...
6
votes
0answers
250 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
6
votes
5answers
2k views

Have any discrete-time continuous-state Markov processes been studied?

I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state ...
5
votes
1answer
280 views

Constructing Martingales from Markov Processes

I know that for a Markov process $X_t$ with generator $L$ and $f,f^2\in D(L)$, $$M_t=f(X_t)-\int_0^t Lf(X_s)\ ds$$ is a martingale (w.r.t. $P^x$). And I want to show that $$M_t^2-\int_0^t (Lf^2(X_s)-...
4
votes
1answer
1k views

Hitting times of Markov chain/process have always finite moments?

Consider an irreducible ergodic Markov chain on a finite state space $S$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state $s \in S$ is ...
3
votes
1answer
598 views

Strong Markov property of Bessel processes

I am thinking about the following: If $(B_t)_{t \geq 0}$ is a Brownian motion in $\mathbb{R}^3$, how can we show that the Bessel process (of order $3$) $(|B_t|)_{t \geq 0}$ has the strong Markov ...
2
votes
2answers
507 views

The strong Markov property with an uncountable index set

The following definition of the strong Markov property, from Klenke's book, supposes an index set $I$ that is not necessarily countable. However, it is explicitly mentioned previously (following Lemma ...
1
vote
3answers
2k views

Estimating the transition matrix given the stationary distribution

Let's say we are given a Markov chain for variable $X = [x_1, ..., x_n]$; also we are given a desired stationary distribution for this graph $P_\infty = [p_1, ..., p_n]^\top$. How can we design an ...
0
votes
4answers
174 views

Expected number of rolls needed to get two consecutive sixes under the condition that all rolls are even

Suppose I keep rolling a die and I stop once I got two consecutive 6. What is the expected number of time to roll the die under the condition that all the rolls are even number? So, the sample space ...
8
votes
2answers
3k views

Interpretation for the determinant of a stochastic matrix?

Is there a probabilistic interpretation for the determinant of a stochastic matrix (i.e. an $n \times n$ matrix whose columns sum to unity)?
5
votes
1answer
171 views

A proof by René Schilling that a continuous Lévy process is integrable

In his treatise "An Introduction to Lévy and Feller Processes" (arXiv link), Prof. Dr. René Schilling gives a short and seemingly straightforward proof for the claim that a continuous Lévy process is ...
5
votes
1answer
216 views

Is a Markov process uniquely determined?

Let $E$ be a Polish space and $\mathcal E$ be the Borel $\sigma$-algebra on $E$ $I\subseteq[0,\infty)$ be closed under addition and $0\in I$ Please consider the following result: Let $(\kappa_t:t\...
4
votes
2answers
148 views

Show that the carré du champ operator is nonnegative

Let $(E,\mathcal E)$ be a measurable space $\mathcal M_b(E,\mathcal E):=\left\{f:E\to\mathbb R\mid f\text{ is bounded and }\mathcal E\text{-measurable}\right\}$ $(\kappa_t)_{t\ge0}$ be a Markov ...
4
votes
1answer
188 views

Building a hidden markov model with an absorbing state.

I'm working on trying to implement a hidden markov model to model the affect of a specific protein that can cut an RNA when the ribosome is translating the RNA slowly. Some brief background: The ...
4
votes
1answer
166 views

Markov property w.r.t. a countable state space

Background Let $\left(X_t\right)_{t \in I}$ ($I\subseteq\mathbb R$) be an $E$-valued stochastic process ($E$ being a Polish space with the Borel $\sigma$-algebra $\mathcal{B}\left(E\right)$) equipped ...
3
votes
2answers
970 views

Markov Chain and Forward and Backward Probabilities with Alice and Bob

System Alice and Bob are moving independently from one city to another. There are $d$ cities, the probability of moving to another city (for each individual) is $m$ and each move is equiprobable (...
3
votes
0answers
43 views
+50

Convergence of Feller processes implies convergence of the resolvent operators of their generators

Let $E$ be a locally compact separable metric space $(T_n(t))_{t\ge0}$ and $(T(t))_{t\ge0}$ be strongly continuous contraction semigroups on $C_0(E)$ with generator $(\mathcal D(A_n),A_n)$ and $(\...
3
votes
1answer
1k views

How to prove periodicity is a class property?

It is said that in a Markov chain, if state $i$ has period $d$ and state $i$ and $j$ are communicate, then state $j$ also has period $d$. I wonder how to prove it?
3
votes
1answer
424 views

Stochastic processes with independent increments

If $\{X_{t}:t\geq 0\}$ is a real-valued stochastic process with independent increments then $\{X_{t}:t\geq0\}$ is a Markov process? Let $\{ \mathcal{F}_{t} \}_{t\geq0} $ be a natural filtration of $\{...
3
votes
2answers
508 views

A Markov process which is not strong Markov process (follow up 2)

In https://mathoverflow.net/questions/43833/a-markov-process-which-is-not-a-strong-markov-process George Lowther's example: "Consider the following continuous Markov process $X$, starting from ...
3
votes
2answers
1k views

Connections between SDE and PDE

I have encountered a number of situations where the solution of a PDE and a certain expectation associated to a Markov process are equal. Two examples include: The heat equation $u_t = \frac{1}{2} \...
2
votes
1answer
122 views

Steps of a Markov chain subordinated to a Poisson process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a time-homogeneous Markov chain on $(\Omega,\mathcal A,\operatorname P)$ $D([0,1]):=\...
2
votes
2answers
556 views

Markovian Gaussian stationary process with continuous paths

Could you, please, help me figure out the following problem. We call a stationary Gaussian process $\xi_t$ (with continuous paths) an Ornstein-Uhlenbeck process if its correlation function $\mathbb{...