Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

3
votes
1answer
62 views

Find a modified coupling $((X_n,\tilde Y_n))_{n∈ℕ_0}$ with the same coupling time $τ$ and $\tilde Y_n=X_n$ for $n≥τ$ in the coupling lemma

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(E,\mathcal E)$ be a measurable space $(X_n)_{n\in\mathbb N_0}$ and $(Y_n)_{n\in\mathbb N_0}$ be independent $(E,\mathcal E)$-valued ...
0
votes
0answers
30 views

If $X$ is strongly Markovian at $\tau$ is $X_{\tau+\;\cdot\;}$ a Markov process?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $I=[0,\infty)$ or $I=\mathbb N_0$ $(\mathcal F_t)_{t\in I}$ be a filtration on $(\Omega,\mathcal A)$ $(E,\mathcal E)$ be a measurable ...
-1
votes
1answer
21 views

If $κ_i$ is a Markov kernel, is there an unique Markov kernel $κ$ on the product space with $κ((x_1,x_2),B_1\times B_2)=κ_1(x_1,B_1)κ_2(x_2,B_2)$? [closed]

Let $(E_i,\mathcal E_i)$ be a measurable space $\kappa_i$ be a Markov kernel on $(E_i,\mathcal E_i)$ Is there an unique Markov kernel $(E_1\times E_2,\mathcal E_1\otimes\mathcal E_2)$ with $$\...
-1
votes
1answer
80 views

Transition Probabilities - Markov Process [closed]

Suppose there is a box with $N$ balls. Each ball is coloured either red or blue. In each time period, one ball is chosen at random from the box and with probability $\frac{1}{2}$ replaced with the ...
1
vote
0answers
45 views

Does a continuous-time stochastic process satisfying this evolution equation have the Markov property?

It is well-known that, for a continuous-time Markov process, the transition probabilities $\mathbf{P} = \mathbb{P}\left[X(t) = j |X(0) = i\right]$ satisfy the following evolution equation: $$\frac{d\...
1
vote
1answer
84 views

Show that i.i.d. process is ergodic

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(E,\mathcal E)$ be a measurable space $(X_n)_{n\in\mathbb N_0}$ be an $(E,\mathcal E)$-valued i.i.d. process on $(\Omega,\mathcal A,\...
6
votes
1answer
198 views

Convergence of the distribution of the Langevin diffusion to its invariant measure

Let $(X_t)_{t\ge0}$ be a solution of $${\rm d}X_t=-h'(X_t){\rm d}t+\sqrt 2W_t,\tag1$$ where $(W_t)_{t\ge0}$ is a Brownian motion and $h$ is such that $X$ is the unique strong solution of $(1)$. Assume ...
3
votes
3answers
149 views

How to understand explosion in continuous-time Markov chains?

It is well known that for example in a pure-birth process, explosion occurs when $$ \sum_{n=1}^\infty \lambda_n^{-1} < \infty $$ where $\lambda_n$ is the birth-rate of a new individual when the ...
0
votes
0answers
51 views

Convergence in total variation distance of Markov kernel $n$-fold composition to the stationary measure

Let $(E,\mathcal E)$ be a measurable space $\mu$ be a measure on $(E,\mathcal E)$ $p:E\to(0,\infty)$ with $$\int p\:{\rm d}\mu=1$$ and $\pi$ denote the measure with density $p$ with respect to $\mu$ $...
0
votes
0answers
41 views

Picking path at random in DAG graph with probability equals to path weight.

I'm refering to the following paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.329.3653&rep=rep1&type=pdf. There is a lemma states: Let $G$ be a directed acyclic graph with ...
1
vote
1answer
63 views

time between transitions in continous time discrete state Markov process

Problem Statement: I want to compute the time between transitions in a birth-death model. As a simple example, consider that individuals are born with rate $\lambda$ and they die at rate $\sigma n$. ...
2
votes
0answers
27 views

Show that the composition of a sub-invariant measure with a sub-Markov kernel is a contraction on $L^p$

Let $(\Omega,\mathcal A)$ be a measurable space $\mu$ be a finite measure on $(\Omega,\mathcal A)$ $\kappa$ be a sub-Markov kernel on $(\Omega,\mathcal A)$ $p\ge1$ I'll denote the composition of $\...
1
vote
0answers
54 views

If $(X_t)_{t\ge0}$ is a Markov process with invariant measure $\mu$, does the distribution of $X_t$ weakly converge to $\mu$ as $t\to\infty$?

Let $(\Omega,\mathcal A,\operatorname P)$ be a complete probability space $(X_t)_{t\ge0}$ be a time-homogeneous Markov process on $(\Omega,\mathcal A,\operatorname P)$ and $\kappa_t$ denote a regular ...
2
votes
0answers
70 views

How to MCMC (or other simulation) given a non-stationary distribution?

Say I was given some directed graph that satisfies the Markov property, has a stationary distribution, $\pi$, and I know the transition probabilities are functions of some unknown parameters $P_{i\to ...
2
votes
0answers
32 views

Find a semigroup, stationary distribution and calculate a probability

Let ${\{X_t | t\ge 0\}}$ be a Markov Process on a state space $S={\{1,2,3\}}$ with a generator $$ G=\begin{bmatrix} -1 & 0 & 1 \\ 3 & -4 & 1 \\ 2 & 0 & -2 \end{...
0
votes
0answers
22 views

Existence of a regular version of the conditional distribution in a specific application

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(\mathcal F_t)_{t\ge0}$ be a filtration on $(\Omega,\mathcal A)$ $(X_t^x)_{(t,\:x)\in[0,\:\infty)\times\mathbb R}$ be a continuous ...
0
votes
0answers
21 views

Proof that thin sets are finely separated

I would like to justify the following claim concerning thin sets found in Chung, Lectures from Markov Processes to Brownian Motion, page 112: Such a set is finely separated in the sense that each ...
1
vote
0answers
24 views

Markov property for unbounded function

Let $(X_t)$ be a Markov process with respect to a filtration $\mathcal{F}_t$. Assume that $P(X_t>0 \, \forall t\geq 0) = 1 $. Denote $E_x$ the expectation under the measure where $X_0=x$. Is it ...
0
votes
1answer
37 views

Markov process: The population distribution of the system after $n$-transitions

I am familiar with discrete markov processes and idea of their convergence towards a stable distribution after some number of steps. But my question is much simpler--more like a validation of some ...
0
votes
0answers
37 views

Residence times of the telegraph process?

The telegraph process is a two state stochastic process defined by the master equation $$ \dot{\pi}_0(t) = \tau^{-1} \pi_1(t) - \sigma^{-1} \pi_0(t) $$ $$ \dot{\pi}_1(t) = \sigma^{-1} \pi_0(t) - \...
-1
votes
1answer
142 views

Invariant measure for Itō diffusion

Let $f\in C^2(\mathbb R)$ be positive and $h\ge 0$. Assume that $g:=f'/f$ is Lipschitz continuous and let $U$ be a strong solution of $${\rm d}U_t=\frac h2g(U_t){\rm d}t+\sqrt h{\rm d}W_t$$ ($W$ being ...
0
votes
0answers
30 views

Number of returns simple random walk $\mathbb{Z}^d$

I am interested to know the precise numerical value of the expected number of returns to the origin of a simple random walk on $\mathbb{Z}^d$, when $d \geq 3$. Does anyone know where I can find such a ...
0
votes
0answers
39 views

Markov Chain first order applied to attribution modelling

Based on this article I'm using within R the Channel Attribution package to leverage on the Markov Chain in order to attribute conversion between several marketing channels. However, the computation ...
1
vote
0answers
40 views

Show $(B_t )^2$ i.e. square of a Brownian motion is a Markov process. [duplicate]

Problem: Show $(B_t )^2$ i.e. square of a Brownian motion is a Markov process. To do this, I want to show $$P( B_t ^2 | B_{t_1} ^2 , ... , B_{t_n} ^2 )= P( B_t ^2 | B_{t_n} ^2 ) $$ where $0<...
1
vote
1answer
21 views

Why is $P^b(B_{T-T_b} \in (-\infty,b))=1/2$

Why is $P^b(B_{T-T_b} \in (-\infty,b))=1/2$ on the set $\{T_b<t\} $ where $T_b=\inf\{t \ge 0 :B_t=b\}$ and $T=t 1_{\{T_b<t\}}+\infty 1_{\{T_b \ge 0\}}$. I am trying to understand Proposition 2....
0
votes
0answers
44 views

When is it necessary to solve Kolmogorov forward equations (KFE) for a Markov Chain?

Say I have a continuous time markov chain, time homogeneous $X$ with a few states (say, 2). I want to know the distribution of where $X$ is at time $t$, call it $\mu_t$, which will be a vector of 2 ...
0
votes
1answer
12 views

Why does $P(Q_t = q | X_{0:L} = i_{0:L}) = P(Q_t = q, X_{0:L} = i_{0:L})$?

This is a derivation of an equation used to maximize the posterior probability that $Q_m = i_m$ given a model and a sequence of observations. $Q_m$ is a RV which maps to some $q \in S$, the state ...
1
vote
1answer
22 views

Conditions for Markov process not to reach point at infinity

My question concerns the book Lectures from Markov Processes to Brownian Motion by Kai Lai Chung, more precisely the remark at the bottom of page 76: We prove later in paragraph 3.3 that on $\{ t &...
1
vote
0answers
58 views

If $Y$ is a Markov chain and $h>0$, why is $(Y_{\lfloor t/h\rfloor})_{t\ge0}$ not a Markov process?

Let $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a Markov chain for $n\in\mathbb N$, $(h_n)_{n\in\mathbb N}\subseteq(0,\infty)$ with $h_n\xrightarrow{n\to\infty}\infty$ and $$X^{(n)}_t:=Y^{(n)}_{\...
3
votes
0answers
92 views

How to find probability transition matrix for continuous time markov chain?

In Grimmet and Stirzaker, on page 258 it explains how to find transition probabilities, given a generator matrix: (a) nothing happens during $(t,t+h)$ with probability $1+g_{ii}*h+o(h)$ (b) ...
2
votes
0answers
63 views

Random walk on $\{0,1,…,k\}$, find the average gain in 10 000 steps

I have the following problem which I can't seem to figure out. The problem is as follows. Consider simple random walk on {0, 1, ... , k} with reflecting boundaries at 0 and k, that is, random ...
1
vote
1answer
45 views

Period of each state

I am trying to determine the period of each state $ j = 0, 1, 2$ for this irreducible Markov Chain with transition probability matrix $$P=\begin{bmatrix}0&0&1\\1&0&0\\\frac{1}{2}&\...
0
votes
0answers
10 views

Why conditional independence assumption in GP models is usually valid? Or it isn't?

I'm interested in the ground for making conditional independence assumption (e.g. that different target dimensions do not covary for given input $x$) when we are modelling some multidimensional signal ...
0
votes
0answers
20 views

Markov property for mixed joint densities

Let $X \rightarrow Y \rightarrow Z$ be a Markov chain in that order, $X$ and $Y$ be jointly Gaussian, and $Z$ be a discrete random variable with finite alphabet $\mathcal{Z}$. Denote their mixed ...
2
votes
1answer
136 views

Steps of a Markov chain subordinated to a Poisson process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $\left(Y^{(n)}_k\right)_{k\in\mathbb N_0}$ be a time-homogeneous Markov chain on $(\Omega,\mathcal A,\operatorname P)$ $D([0,1]):=\...
2
votes
0answers
53 views

Backward Kolmogorov equation for simple markov process

The following exercise is from a course on SDE's and I am a bit stumped. Consider the process. $dX_t=\lambda\left(\xi-X_t \right)dt+\gamma\sqrt{|X_t|}dB_t$ $\lambda,\xi,\gamma>0$ Find $\mathbb{P}...
1
vote
1answer
83 views

Transition rate matrix of a combined birth-death processes.

If the transition rate matrix X of one birth-death process is defined by \begin{bmatrix} -\lambda & \mu \\ \mu & -\lambda \end{bmatrix} and another transition rate matrix Y of a second birth-...
0
votes
0answers
21 views

How do I interpret the following $E^{X_S}[f(X_t)]$?

Given a Markov family $X=\{X_t, \mathcal{F}_t, t\ge 0\}$ on some $(\Omega,\mathcal{F})$,together with a family of probability measures $\{P^x\}_{x \in \mathbb{R}^d}$ on $(\Omega,\mathcal{F})$ and a ...
1
vote
0answers
59 views

Branching process and posterior distribution.

I have the following branching process: (I added the $Z$'s myself for the number of individuals in each gen, hope it's correct.) Assume the offspring distribution is Poisson with expectation $\...
1
vote
1answer
40 views

Stopping time of Feller process

Let $X$ be a Feller process on $\mathbb{R}$ with generator $Gf=\frac{1}{2}f''-f'$ on $C_c^2$. Let $\tau_b$ be the first time that $X$ hits $b\in\mathbb{R}$. Show that for $x > 0$, $bP_x(\tau_b < ...
0
votes
1answer
59 views

What exactly does it mean to multiply a vector by the transition matrix of a Markov process?

I know that given a stationary distribution and 2 state transition matrix that $\begin{pmatrix} \Pi _{1} & \Pi _{2} \end{pmatrix}\begin{pmatrix} P_{00} & P_{01}\\ P_{10}& P_{11} \end{...
1
vote
1answer
29 views

Probability that a random walk does not leave a subgraph after $k$ steps

Suppose I have a connected, finite, graph $G = (V,E)$, and I have some vertex set $U$ such that the subgraph of $G$ induced by the vertices $Y = V \setminus U$ is still connected. Now, suppose I have ...
0
votes
0answers
13 views

Alternative formulation of a markov process

I'm wondering how the markov property can be specified as follows, if anyone can provide more details (this looks awfully like the definition for a martingale): $$E[f(X_t)|\mathcal{F}_s]=E[f(X_t)|\...
1
vote
1answer
23 views

Why $P^\mu[X_{S+t} \in \Gamma \mid \mathcal{F}_{S+}]=P^\mu[X_{S+t} \in \Gamma \mid X_{S}]=0 \text{ ,} P^\mu\text{-a.s. on } \{S=\infty\}$

For any progressively measurable process $X$ and any optional time $S$ of $\{\mathcal{F}_t\}$ why do we have that $$P^\mu[X_{S+t} \in \Gamma \mid \mathcal{F}_{S+}]=P^\mu[X_{S+t} \in \Gamma \mid X_{S}]=...
5
votes
1answer
537 views

Convergence of discrete-time Markov chain to Feller processes

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(X_t)_{t\ge0}$ be a Feller process on $(\Omega,\mathcal A,\operatorname P)$ $(h_d)_{d\in\mathbb N}\subseteq(0,\infty)$ with $$h_d\...
0
votes
0answers
57 views

Backward Kolmogorov equation to find probability

From lecture notes in a course on SDE's. We are tasked with using the backward Kolmogorov equation to find. $\mathbb{P}^{X_t=x}\left(X_T\geq2 \right)$ I am confused by the terminology here. We are ...
0
votes
0answers
140 views

Birth-death process Expected waiting time in when in invariant distribution

Say I have a birth-death process, with birth having Poisson distribution with parameter $\lambda$ and death having poisson distribution $\mu$. Assuming that both stochastic processes, birth and death, ...
0
votes
1answer
23 views

A question on the defintion of Markov process

Included in the defintion of the Markov process is the following $\text{ for } x \in \mathbb{R}^d,s,t \ge 0, \Gamma \in \mathcal{B}(\mathbb{R}^d)$ $$, P^x[X_{t+s} \in \Gamma \mid X_s=y]=P^y[X_t \in \...
0
votes
0answers
14 views

Choosing between non-determinism and probabilistic models

I have a stochastic system such that there are discrete states. At each discrete state, one or more probabilistic transition rules apply. For example, when Sam is in house he can go to school with a ...
0
votes
0answers
79 views

Question related to Discrete Markov Chains

Consider a dual core processor which operates using a cache memory. Two processes cannot simultaneously access the cache i.e. if a process is accessing the cache and another process tries to access it,...