Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

1
vote
0answers
10 views

Exponential moment of return times markov process

For a discrete time irreducible + aperiodic markov chain (with general state space X) the geometric ergodicity criteria implies exponential return times to compact sets. More preciesly: The geometric ...
0
votes
1answer
29 views

Why are Markov processes are completely determined by their initial value?

I'm reading a probability theory book, which (slightly reworded) says the following: A Markov process is completely determined once we know $$P_{ij}^{n,n+1} = P\{X_{n + 1} = j \mid X_{n} = i\} $$...
1
vote
1answer
23 views

Sufficient condition for a Markov chain to be Aperiodic

If I want to prove that a Markov chain is aperiodic, then if I can show that $P(X_{n+1}=i\mid X_n=i)\gt 0$ $ \forall i$. Then can I say that the chain is aperiodic?
1
vote
0answers
15 views

Markov property of SDE's solution

Considering the SDE $dX_t=b(t,X_t)dt+\sigma (t,X_t)dW_t$ ($W$ is Brownian motion)  If there exists weak solution $(X,W),(\Omega ,\mathscr{F} ,P),\{\mathscr{F}_t\}$, is $X$ Markov process? I know ...
0
votes
0answers
12 views

Existence of recurrent and transient classes in general state Markov chains

For Markov chains defined over countably finite-state spaces, it has been proven that recurrent and transient classes exist [Durrett, Richard, "Essentials of stochastic processes",Springer,1999]. I ...
0
votes
1answer
16 views

Varying definitions of a martingale

I frequently see that, over some filtration $\mathscr{F}_{n}, \{X_{n}\}$ is defined as a martingale if $E[X_{n+1}|\mathscr{F}_{n}]=X_{n}$. Sometimes, however, I see this extended to $E[X_{n+s}|\...
0
votes
1answer
23 views

Infinitesimal generator/intensity matrix

I am trying to understand the following example, but I do not understand how they get $P(t)$. What are they doing with the diagonal matrix to get the three matrices with $e^{-t}$, $e^{-3t}$ and $e^{-...
1
vote
0answers
15 views

Markov process intensity matrix

$X$ is a Markov process with state space $(1,2,3)$. How can I find the matrices of transition probabilities $P(t)$ if the generator is \begin{bmatrix}-2&2&0\\2&-4&2\\0&2&-2\end{...
0
votes
0answers
25 views

Showing $f_n(f(e^{-C_{n+1}^{-1}})^z) = f_{n+1}(e^{-zC_{n+1}^{-1} })$ where $f_n$ is the pgf of a Galton-Watson process $Z_n$

I want to show $$f_n(f(e^{-C_{n+1}^{-1}})^z) = f_{n+1}(e^{-zC_{n+1}^{-1} })$$ In this case $Re(z)\ge 0$, $(C_n)$ is a sequence of constants and $f_n$ is the probability generating function of $Z_n$. ...
1
vote
0answers
27 views

Computing a Diffusion Limit of a Markov Chain

Fix $\alpha >0$, and for $h > 0$, consider the Markov kernel $K_h$ derived by composing the following two `moves': From $x^t$, move to $x^{t+1/2} = x^t + y^t$, where $y^t \sim \text{Gamma}(h, 1)...
0
votes
0answers
12 views

Reflected Lévy process is a Markov process

Let $L=(L_{t},t\geq 0)$ be a Lévy process on $(\Omega,\mathcal{F}, \mathbb{P})$ , $\mathbb{P}_{x}$ the distribution of $L+x$ under $\mathbb{P}$ and let $\mathbb{F}=(\mathcal{F}_{t}, t\geq 0)$ the ...
1
vote
0answers
36 views

If $L$ is a diffusion operator with corresponding carré du champ operator $\Gamma$, then $Lf^2=2fLf+2\Gamma(f)$

Let $(E,\mathcal E,\mu)$ be a measure space and $$\mu f:=\int f\:{\rm d}\mu\;\;\;\text{for }f\in L^1(\mu)$$ $\mathcal A_0$ be a subspace of $\left\{f:E\to\mathbb R\mid f\text{ is bounded and }\...
1
vote
1answer
58 views

Extending the domain of the Dirichlet form associated with a symmetric Markov semigroup

Let $(E,\mathcal E)$ be a measurable space $\mathcal M_b(E,\mathcal E):=\left\{f:E\to\mathbb R\mid f\text{ is bounded and }\mathcal E\text{-measurable}\right\}$ $(P_t)_{t\ge0}$ be a Markov semigroup ...
1
vote
0answers
31 views

Value function optimization problem

How to show that V* (optimal value function), is the solution to the following optimization problem: $min_V~ \Sigma_s V(s) $ with the constraint: $ V \ge T^{*}V $ Where $T^{*}$ is the optimal Bellman ...
0
votes
1answer
15 views

If a transition matrix $A$ is regular, prove that $A^∞$ has all the same columns and that the columns are the steady state vector.

I know that this is quite an elementary theorem, but I have yet to see a proof of this except for quoting that $A^∞$ is going to be of rank 1, so that all the columns must be the same. Any help is ...
4
votes
2answers
84 views

Show that the carré du champ operator is nonnegative

Let $(E,\mathcal E)$ be a measurable space $\mathcal M_b(E,\mathcal E):=\left\{f:E\to\mathbb R\mid f\text{ is bounded and }\mathcal E\text{-measurable}\right\}$ $(\kappa_t)_{t\ge0}$ be a Markov ...
1
vote
0answers
54 views

Show that $C_c^∞(\mathbb R)$ is a core of the generator of the Feller semigroup induced by the strong solution of an SDE with Lipschitz coefficients

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $b,\sigma:\mathbb R\to\mathbb R$ be Lipschitz continuous and $$Lf:=bf'+\frac12\sigma^2f''\;\;\;\text{for }f\in C^2(\mathbb R)$$ $W$ be ...
1
vote
1answer
49 views

Is the transition semigroup of the solution of an SDE with Lipschitz coefficients strongly continuous on $C_b$?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $b,\sigma:\mathbb R\to\mathbb R$ be Lipschitz continuous (and hence at most of linear growth) and $$Lf:=bf'+\frac12\sigma^2f''\;\;\;\...
0
votes
1answer
16 views

How to formulate a queueing model where riders and drivers arrive randomly and are matched at every specific interval?

I am going to develop a queueing model in which riders and drivers arrive with inter-arrival time exponentially distributed. All the riders and drivers arriving in the system will wait for some ...
4
votes
2answers
60 views

A strange inconsistency between a calculation and a simulation of a TASEP

Consider the following stochastic process, called a totally asymmetric simple exclusion process (TASEP), on the integers $\mathbb{Z}$: The process evolves over discrete time steps $T = 1, 2, \ldots \...
-1
votes
1answer
23 views

Why the following set of eigenvalues will not work for any allowed values of a and b? [closed]

Following markov chain has transition matrix: $ \begin{pmatrix} 0.7&a&0.3-a\\b&0.5-b&0.5\\1&0&0\end{pmatrix} $ Explain why the following set of eigenvalues is impossible for ...
0
votes
1answer
34 views

Calculating return time / showing null recurrence of discrete Markov chain

Recently I have been attempting a question on continuous time Markov chains, and one of the parts prompts me to verify the null recurrence of a CTMC's jump chain, which is a discrete Markov chain. ...
2
votes
1answer
40 views

Classes and Markov Chains

The Markov chain $(Xn; n\geq)$ has state-space $S = (0, 1, 2, . . .)$, with $p_{i,0} = \frac{1}{4}$ and $p_{i,i+1} = \frac{3}{4}$ $\forall i \geq 0$, so that the transition matrix is P =$\...
0
votes
1answer
41 views

What is wrong with this solution of the extinction problem?

I am considering a standard extinction problem, that is: A bacterial colony consists of individual bacteria. One of the following happens with each bacterium each second: The bacterium dies. The ...
2
votes
1answer
39 views

Derivation of the Fokker-Planck equation

Let $b\in C^1(\mathbb R)$ be Lipschitz continuous $\sigma\in C^2(\mathbb R)$ be Lipschitz continuous with $\sigma(\mathbb R)\subseteq\mathbb R\setminus\left\{0\right\}$ and $\sigma''$ being bounded $...
1
vote
1answer
97 views

Bound error due to approximation by Markov chain

Basic setting I am working with a sequence of random variables $\mathbf{X} := X_1, X_2, \dots$, for which I know the Markov property does not hold exactly, but approximately: $$ \Pr[X_{n+1}=x \mid X_{...
-1
votes
1answer
35 views

Markov chain duration [closed]

What is the formula to find average duration of state s in a Markov chain given a transition matrix? I tried to recall the concept but could not find any references.
0
votes
0answers
15 views

Product Rule for Expectation

Denote $f$ and $g$ two functions of $x_t$, where $x_t$ is a Markov diffusion: $$ dx_t = \mu(x_t)dt + \sigma(x_t)dZ_t $$ For $T \geq 0$, I would like to give some intuition about what drives the ...
3
votes
0answers
47 views

Show that if $(κ_t)_{t≥0}$ is the transition semigroup of a strong solution to an SDE, $t↦(κ_tf)(x)$ is continuous for all $x$ and suitable $f:ℝ→ℝ$

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $b,\sigma:\mathbb R\to\mathbb R$ be Lipschitz continuous and $$Lf:=bf'+\frac12\sigma^2f''\;\;\;\text{for }f\in C^2(\mathbb R)$$ $W$ be ...
0
votes
0answers
18 views

Large, sparse transition matrix - strategies for computing given memory limitations

Beyond some size of the transition matrix, my computer cannot cope with the Markov Chain problem I am working on (in MATLAB). However, I am sure I am not aware of all the useful tricks that can extend ...
3
votes
1answer
39 views

Show that two Markov kernels almost surely agree

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(E_i,\mathcal E_i)$ be a measurable space $X_1:\Omega\to E_1$ $X_2:\Omega\to E_2$ be $(\mathcal A,\mathcal E_2)$-measurable $\kappa$ ...
1
vote
1answer
86 views

If $(κ_t)_{t≥0}$ is the transition semigroup of a continuous Markov process, is $t↦(κ_tf)(x)$ continuous for all bounded continuous $f$ and fixed $x$?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(\mathcal F_t)_{t\ge0}$ be a filtration on $(\Omega,\mathcal A)$ $E$ be a metric space $(X_t)_{t\ge0}$ be an $E$-valued right-...
3
votes
1answer
77 views

Why is it legitimate to assume that the Chapman-Kolmogorov equations hold everywhere?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(\mathcal F_t)_{t\ge0}$ be a filtration on $(\Omega,\mathcal A)$ $E$ be a Polish space and $\mathcal E:=\mathcal B(E)$ $(X_t)_{t\ge0}$...
1
vote
0answers
32 views

Fokker-Planck equation for a Markov semigroup with densities

Let $(E,\mathcal E)$ be a measurable space $(\kappa_t)_{t\ge0}$ be a Markov semigroup on $(E,\mathcal E)$ and $$\kappa_tf:=\int\kappa_t(\;\cdot\;,{\rm d}y)f(y)$$ for $$f\in F_0:=\left\{f:E\to\mathbb ...
0
votes
0answers
8 views

Stuck with a limit calculation for a Markov Decision Process.

I'm trying to compute the following limit from a Markov Decision Process exercise (namely 10.7, from Sutton, RL An Introduction). I formulated the problem as $v_{\pi}(A) = \lim\limits_{\gamma \...
0
votes
0answers
23 views

Does a given First Passage Time Distribution imply a single possible Fokker-Planck equation?

Consider a 1-dimensional Continuous Markov Process $X(t)$ with fixed and constant absorbing boundaries, let's say at $\pm \theta$, and with starting point at $X(t=0)=0$. This setting will lead to a ...
1
vote
1answer
74 views

Markov chain converges to Normal Distribution. How to increase standard deviation

I am creating an MC (using the following recursive function). It is about a game that each side has different probs of scoring 1 point, 2 points, 3 points or no score (scoreDifference is the score ...
0
votes
1answer
16 views

Equivalence of discrete definition of Markov property in the coutinuous case

In the book, Lectures from Markov Processes to Brownian Motion, it is stated that the oldest definition of Markov property is, for every integer $n\ge1$ and $0\le t_1<t_2<\cdots<t<u,$ and $...
3
votes
0answers
15 views

Question about integral notation in a Markov process + how to evaluate said integral

I'm reading Chapter 11 of Puterman's book on Markov Decision Processes (in particular, about continuous-time Markov processes). There's a lot of notation involved, but I've tried to distill the ...
0
votes
0answers
18 views

What does it mean that “general states are not conserving probability so, it could be neither a quantum system, nor a Markov chain”?

I read from here Since $\hat{H}$ is Hermitian matrix, its eigenvalues are real numbers. Components with positive eigenvalues decrease to zero and negative eigenvalues blow up to infinity. The only ...
0
votes
0answers
12 views

How can we show that this is a Markov kernel?

Let $(E,\mathcal E)$ be a measurable space $\lambda$ be a measure on $(E,\mathcal E)$ $p:E\to(0,\infty)$ be $\mathcal E$-measurable $q:E\times E\to(0,\infty)$ be $\mathcal E\otimes\mathcal E$-...
0
votes
0answers
18 views

Show that for each Markov kernel admiting a density there is a corresponding reverse kernel

Let $(E_i,\mathcal E_i)$ be a measurable space $\kappa$ be a Markov kernel with source $(E_1,\mathcal E_1)$ and target $(E_2,\mathcal E_2)$ $\mu$ be a probability measure on $(E_1,\mathcal E_1)$ ...
1
vote
0answers
36 views

Show that two processes have the same distribution knowing that their paths up to time $τ$ and their distributions conditioned on $\mathcal F_τ$ agree

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(E,\mathcal E)$ be a measurable space $(Y_n)_{n\in\mathbb N_0}$ and $(\tilde Y_n)_{n\in\mathbb N_0}$ be time-homogeneous Markov ...
1
vote
0answers
41 views

Reverse kernel of a Markov kernel with density

Let $(E_i,\mathcal E_i)$ be a measurable space $\kappa$ be a Markov kernel with source $(E_1,\mathcal E_1)$ and target $(E_2,\mathcal E_2)$ Assume $\kappa$ has a positive density with respect to a ...
0
votes
0answers
43 views

Find the long-term mean number of cars in line at the toll booth.

Cars arrive at a toll booth according to a Poisson process at the rate of two cars per minute. The time taken by the attendant to collect the toll is exponentially distributed with mean $20$ ...
3
votes
1answer
45 views

Find a modified coupling $((X_n,\tilde Y_n))_{n∈ℕ_0}$ with the same coupling time $τ$ and $\tilde Y_n=X_n$ for $n≥τ$ in the coupling lemma

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(E,\mathcal E)$ be a measurable space $(X_n)_{n\in\mathbb N_0}$ and $(Y_n)_{n\in\mathbb N_0}$ be independent $(E,\mathcal E)$-valued ...
0
votes
0answers
25 views

If $X$ is strongly Markovian at $\tau$ is $X_{\tau+\;\cdot\;}$ a Markov process?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $I=[0,\infty)$ or $I=\mathbb N_0$ $(\mathcal F_t)_{t\in I}$ be a filtration on $(\Omega,\mathcal A)$ $(E,\mathcal E)$ be a measurable ...
-1
votes
1answer
20 views

If $κ_i$ is a Markov kernel, is there an unique Markov kernel $κ$ on the product space with $κ((x_1,x_2),B_1\times B_2)=κ_1(x_1,B_1)κ_2(x_2,B_2)$? [closed]

Let $(E_i,\mathcal E_i)$ be a measurable space $\kappa_i$ be a Markov kernel on $(E_i,\mathcal E_i)$ Is there an unique Markov kernel $(E_1\times E_2,\mathcal E_1\otimes\mathcal E_2)$ with $$\...
-1
votes
1answer
66 views

Transition Probabilities - Markov Process [closed]

Suppose there is a box with $N$ balls. Each ball is coloured either red or blue. In each time period, one ball is chosen at random from the box and with probability $\frac{1}{2}$ replaced with the ...
1
vote
0answers
44 views

Does a continuous-time stochastic process satisfying this evolution equation have the Markov property?

It is well-known that, for a continuous-time Markov process, the transition probabilities $\mathbf{P} = \mathbb{P}\left[X(t) = j |X(0) = i\right]$ satisfy the following evolution equation: $$\frac{d\...