Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Questions on the Kantorovich-Rubinstein duality

Let $\mu,\nu$ be probability measures on a metric space $(E,d)$ endowed with the Borel $\sigma$-algebra and $$\operatorname W_d(\mu,\nu):=\inf_{\gamma\in\mathcal C(\mu,\:\nu)}\int d\:{\rm d}\gamma,$$ ...
0
votes
0answers
8 views

When does a Markov semigroup preserve differentiability?

Let $E$ be a $\mathbb R$-Banach space (for simplicity, assume $E=\mathbb R$, if you like) and $(\kappa_t)_{t\ge0}$ be a Markov semigroup on $(E,\mathcal B(E))$. I would like to know under which ...
-1
votes
1answer
28 views

About measures acting on measurable functions.

If $\phi : X \rightarrow \mathbb{R}$ is a measurable function and $\mu$ is a measure on $X$ then what is $``\mu(\phi)"$? Is this a notation for some function? I came across this notation for the ...
0
votes
0answers
19 views

The quadratic variation of the Brownian motion almost certainly tends to T

On the segment $[0, T]$, choose $n$ independent points $t_{n,k}$ (each distributed evenly). Prove that the quadratic variation of the Brownian motion on the sequence of random partitions of the ...
0
votes
2answers
41 views

How to prove that Markov chain with specific transition probabilities has independent increments?

I have Markov chain $N=\{N(t) \mid t\geq 0 \}$ with the state space $\{0,1,2,\dots\}$. I know that it is homogeneous and transition probabilities are: $$ p_{ij}(s,t)=P(N(t)=j\mid N(s)=i) = p_{i,j}(t-...
1
vote
0answers
26 views

Consider a branching process $\{X_n , n \geq 0 \}$ in which the offspring distribution is binomial $(k,p)$

Consider a branching process $\{ X_n , n \geq 0 \}$ in which the offspring distribution is binomial $(k,p)$. Find probability of ultimate extinction when $k = 3$. So I've tried this: $P_k = P(\...
0
votes
0answers
16 views

How do you find the probability of a continuous-time markov chain, where having started in state $i$, will be in $j$ state at time $t$?

This is the three-state continuous-time Markov chain in which the transition rates are given by: $$Q = \left[ \begin{matrix} 0 & 2\lambda & 0 \\ \lambda & 0 & \lambda \\ ...
0
votes
0answers
13 views

ergodicity of non-linear stochastic dynamical system

Is there any ergodicity result( existance and uniqueness of invariant probability measure) for the Random dynamical system which form a discrete-time markov chain by its realization equation as: $y_{t+...
1
vote
0answers
28 views

Determine the mean and variance of a branching process.

Determine the mean and variance of a branching process with offspring in which an individual either dies, with probability $p$, or is replaced by two progeny, with probability $1−p$. Would this be a ...
0
votes
0answers
11 views

Every finite closed class is recurrent proof clarification

In Norris: Markov Chains the closed class C is defined as one for which $i\in C$ and $P_i(X_n=j \text{ for some }n\ge0)>0$ implies that $j\in C$. Here's theorem 1.5.6 from the book with proof ...
0
votes
0answers
8 views

Measurability of the exterior boundary of a set for a transition kernel

I am learning the potential theory for the Markov chains. And I've encountered a problem: Let $\pi$ be a transition kernel on a Polish space $(S,\mathcal{B})$, and let $D \in \mathcal{B}$. The ...
1
vote
1answer
25 views

Markov chain with external input

Could anyone explain to me this Markov chain model? $$S_{k+1}= P(S_k+S_k^0).$$ Please allow me to give a link from the paper I was read this equation $(6)$ here https://drive.google.com/file/d/...
0
votes
0answers
13 views

Non-linear Markov chain model [duplicate]

Could anyone explain to me this Markov chain model? $$S_{k+1}= P(S_k+S_k^0).$$ Please allow me to give a link from the paper I was read this equation $(6)$ here https://drive.google.com/file/d/...
0
votes
0answers
13 views

Every Lévy Process satisfies the markov property?

Is it true that if $\{X_t\}_{t\geq0}$ is a Lévy Process then it also satisfy the markov property?
3
votes
1answer
32 views

$\mathbb P(\sup_{t\in[0,1]}|W_t|\le1)$ for Brownian motion

What is $\mathbb P(\sup_{t\in[0,1]}|W_t|\le1)$ for $W_t$ a Brownian motion? Without the absolute value, we have $\mathbb P(\sup_{t\in[0,1]}W_t\le c)=1-\sqrt{2/\pi}\int_c^\infty e^{-x^2/2}dx$ for all $...
0
votes
0answers
50 views

Gamblers ruin problem where I got the same answer for section e and f. My answer for Alice's Probability of winning is 0.02030134814

Consider the gambler’s ruin problem. Alice starts with £a; her opponent Bob starts with £(m − a). In each round, Alice wins £1 from Bob with probability p = 0.4, or loses £1 to Bob with probability 1 −...
0
votes
0answers
30 views

Markov Matrix with complex eigenvalues

What properties does a Markov matrix (with real entries) with complex eigenvalues have? For example, consider this matrix: $$\begin{pmatrix} 0 & 0 & 1\\ 1 & 0 & 0\\ 0 & 1 & 0\\...
1
vote
0answers
23 views

Multiple Random Walkers that reflect each other

Say I have $2$ random walkers that are positioned along a line. One of them will be on the left and the other on the right. Every time they meet, this ordering does not change, namely the two walkers ...
0
votes
0answers
10 views

MDP tabular setting.

I would be very curious to know if in the tabular MDP setting we have: $$ E_{\tau_1,\cdots, \tau_{N} \sim P^{\pi}_{\mu}} (1_{A}) = E_{s_1,\cdots,s_{N \times H} \sim d^{\pi}_{\mu}} E_{a_1 \sim \pi(.|...
1
vote
0answers
7 views

Markov decision Processes - Optimal state value function

I want to know how an optimal state value function defined for Markov decision Processes Could anyone be kind enough to define the Optimal State value function for MDP?
-3
votes
1answer
50 views

Question on circular random walk [closed]

A truck transports goods among $10$ points located on a circular route. These goods are carried only from one point to the next with probability $p$, or to the preceding point with probability $q=1-p$....
2
votes
1answer
31 views

Let $(X_t)$ be a continuous-time Markov chain and $\tau$ the first jump time. Compute $\mathbb E_x [a^{\tau} \phi (X_\tau)]$

Let $(X_t)$ be a continuous-time Markov chain such that The state space $V$ is finite and endowed with discrete topology. The infintesimal generator is $L: V^2 \to \mathbb R$. Let $\alpha \in (0,1)$...
1
vote
1answer
22 views

How to get $\mathbb E[a^{\tau_1} \phi(X_{\tau_1}) | X_0 =x] = \mathbb E[a^{\tau_2} \phi(X_{\tau_2}) | X_0 =x]$ from Strong Markov property?

Consider a continuous-time Markov chain $(X_t)_{t \ge 0}$ with respect to a completed right-continuous filtration $(\mathcal G_t)_{t \ge 0}$. Suppose that The state space $V$ is finite and endowed ...
1
vote
2answers
23 views

Does this martingale have right-continuous (or cadlag) sample paths?

Let $(X_t)$ be an irreducible continuous-time Markov chain $(X_t)_{t \ge 0}$ with respect to its canonical filtration $(\mathcal G_t)_{t \ge 0}$. Suppose $(\mathcal G_t)_{t \ge 0}$ is completed and ...
0
votes
0answers
9 views

Does irreducibility imply this stopping time is almost surely finite?

Let $(X_t)_{t \ge 0}$ be an irreducible continuous-time Markov chain with finite state space $V$. Let $D \subseteq V$ be open and consider the stopping time $T = \inf \{t \ge 0 \mid X_t \in D\}$. The ...
0
votes
1answer
23 views

Show that this Markov chain has infnitely many stationary distributions and give an example of one of them.

We have the following Markov chain with labelled probabilities. I have constructed the 1-step transition matrix of this Markov chain $$P=\begin{bmatrix}1&0&0&0&0\\0.6&0&0.4&...
0
votes
0answers
13 views

Nonlinear Markov Process bounds

Let $\mathcal{S} \subseteq \mathbb{R}^n$ and $\mu_0 \in \mathcal{P}(\mathcal{S})$ where $\mathcal{P}(\mathcal{S})$ is the set of probability measures on $\mathcal{S}$. I have two discrete-time non-...
0
votes
0answers
5 views

Bessel bridge and process - transition density and SDE

I am somehow confused about what is going on here on page 186-187: https://books.google.de/books?id=7vZ0DwAAQBAJ&lpg=PA111&ots=Q-CelGtKTX&dq=transition%20dimensional%20bessel%20bridge&...
0
votes
0answers
18 views

How to efficiently take the logarithm of a sum of products?

I'm trying to compute the forward procedure on an HMM. While my code works just fine for small input sizes, a long chain leads to probabilities very close to 0 (as is to be expected.) However I later ...
1
vote
0answers
18 views

Reference request: Langevin Dynamics

I'm interested in understanding papers on sampling from a probability distribution via Langevin dynamics as for example: Sampling from a log-concave distribution with Projected Langevin Monte Carlo ...
1
vote
1answer
73 views

Markov chain Monte Carlo with stopping time

I'm doing my thesis where I'm required to compute the numerical value in the following problem: Let $(X_t)$ be a continuous-time Markov chain such that $X_0 = a$ almost surely. The state space $V$ ...
0
votes
0answers
13 views

Birth Rate Question in Stochastic process

Della has four tasks that need to be completed, which must be performed in order. The times taken to perform each task are independent exponentially distributed random variables. He expects that on ...
0
votes
1answer
43 views

Version of the local central limit theorem for a simple random walk on $\mathbb{Z}^D$

Show that for the simple random walk on $\mathbb{Z}^D$ with $D\in\mathbb{N}$ one has $(P^{2n})_{0,0}\cdot D^{\frac{n}{2}}\overset{n\to\infty}{\to}const.\cdot D$ by taking the following steps. (i) ...
0
votes
0answers
13 views

Can someone explain me how forward algorithm works in a Coupled Hidden Markov Model?

I am currently trying to implement, in Python, the forward algorithm of Coupled HMM for modeling two interacting sequences. The aim is actually to use MCMC sampling with forward algorithm to filter ...
1
vote
0answers
8 views

Simplifying the difference of Dirichlet forms of a Markov kernel composed with an involution

Let $(E,\mathscr{E},\mu)$ be a measure space and $P$ be a Markov kernel : $E \times \mathscr{E} \to [0,1]$. For $f \in L^2(\mu)$ define the Dirichlet form $$\mathcal{E}(f,P):= \langle f, (Id-P)f \...
1
vote
0answers
17 views

Exit time of a jump prcess

Let $(X_t)_t$ be a jump Markov process with a symmetric Kernel $J:\mathbb R\times \mathbb R\to \mathbb R^+$. $J(x, y)>0$ gives the intensity of jumping from $x$ to $y$. Let $D$ be a region in $\...
0
votes
0answers
9 views

Markov Chain) infinity of probability process

Consider a Markov Chain {$X_n$;$n$ = $0, 1, ...$}, specified by following transition diagram enter image description here the markov chain($P$) is as follow: $$P=\begin{bmatrix} 0.6&0.4&0&...
1
vote
1answer
27 views

Is jump time in a continuous-time Markov chain a stopping time?

Assume $V$ is a countable state space and $L:V^2 \to \mathbb R$ the infinitesimal generator of a continuous-time Markov chain $(X_t)_{t \ge 0}$ on the probability space $(\Omega, \mathcal{G}, \mathbb{...
0
votes
2answers
43 views

Properties of a Poisson Distribution

So if we were to say $X(t)$ is a Poisson process with rate $λ$, I'm trying to understand this idea: If we fix $n$ then we would expect the time between the $n^{th}$ arrival and the $(n+1)^{th}$ ...
0
votes
0answers
7 views

Conditions for optimal stationary strategies in MDPs

I have a specific markov decision process which is generated from a problem in another domain. What I would like to show is that under the limit of means criterion (no discounting) the optimal ...
0
votes
0answers
27 views

Simulation of Hard Core Model

i already learned about Hard core Model and i know the steps Using markov chain monte carlo (MCMC) to realise it.But i don't know how to make/do a Simulation of Hard code Model in practice. Can ...
1
vote
1answer
23 views

Invariant of reversible Markov chain

Let $\{X_n\}$ be a Markov chain with state space $\mathbb{X}$ and stochastic matrix $P$. If $\{X_n\}$ is irreducible and $\mathbb{X}$ is finite, then it has a unique invariant. My question is: can ...
0
votes
1answer
22 views

Is $I(\omega_0) = J(\omega_0)$ for this homogeneous continuous-time Markov chain?

Assume $(X_t)_{t\ge 0}$ is a homogeneous continuous-time Markov chain on the probability space $(\Omega, \mathcal{G}, \mathbb{P})$. Moreover, $X_1(\omega_0) = X_2(\omega_0)$ for some $\omega_0 \in \...
0
votes
1answer
27 views

How is time until transition defined in this lecture note of continuous-time Markov chain?

I'm reading lecture note Continuous time Markov chains by Alejandro Ribeiro: Continuous time positive variable $t \in \mathbb{R}^{+}$. States $X(t)$ taking values in countable set, e.g., $0,1, \ldots,...
0
votes
1answer
27 views

What does it mean by “the process starts out in state $i$”?

Let $\{X(t): t \geq 0\}$ be a continuous-time Markov chain. Let $T_{i}$ denote the holding time in state $i$. Then we have the following proposition: Could you please explain what does it mean ...
0
votes
1answer
20 views

Proof surrounding Markov Chains

I have come across this question and am fairly unsure how to demonstrate this proof: Prove or provide an explicit counterexample. If a Markov chain on a state space $S$ has a state $i$ with period $...
1
vote
1answer
42 views

How to recover the continuous Markov chain from its infinitesimal generator?

Assume $V$ is a countable state space and $L:V^2 \to \mathbb R$ the infinitesimal generator, and $\mu$ the initial distribution. Moreover, $(X_t)_{t \ge 0}$ is the associated continuous Markov chain ...
0
votes
1answer
29 views

Given the infinitesimal generator, how a continuous Markov chain behaves after the exploding time?

I'm reading paper Explosion, implosion, and moments of passage times for continuous-time Markov chains: a semimartingale approach: Let $\mathbb X$ be the state space and $\Gamma=(\Gamma_{x y})_{x, y \...
0
votes
1answer
36 views

Gamblers Ruin Problem

I'm faced with this problem which is a slight variation on the classic Gamblers Ruin problem: Adam always starts with £$2$ and Bella with £$14$. At each point of the game, Adam can bet any integer ...
0
votes
2answers
44 views

Show $1$ is a persistent null state in $\left(\begin{smallmatrix}1/2&1/2&0\\0&0&1\\\frac{1}{n+1}&0 &\frac{n}{n+1}\end{smallmatrix}\right)$

A three-state time-inhomogeneous Markov chain has the transition matrix: $$\left(\begin{array}{ccc} 1/2 & 1/2 & 0 \\ 0 & 0 & 1 \\ \frac{1}{n+1} &0 &\frac{n}{n+1} & \end{...

1
2 3 4 5
37