Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

-2
votes
0answers
22 views

Prove periodicity is a class property

Prove that if state $i$ in a class has period $p$ then all states in that class have period $p$. The proof is given on this answer is this: One way to define the period of state $i$ is as the ...
0
votes
1answer
8 views

Explicit formula for the one-dimensional distributions of a time-homogeneous Markov chain subordinated by a Poisson process

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space, $(E,\mathcal E)$ be a measurable space, $(X_n)_{n\in\mathbb N_0}$ be a $(E,\mathcal E)$-valued time-homogeneous Markov chain on $(\...
0
votes
0answers
7 views

Markovian Hawkes Process elementary proof

In the book An Introduction to the Theory of Point Processes I by Vere-Jones exercise 7.2.5 asks to show that the intensity of a Hawkes process with exponential intensity kernel is Markov. I found ...
1
vote
0answers
13 views

If a strong Markov process reaches a Borel set a.s., can it be restarted from that set?

Let $X$ be a strong Markov process on $E$, and $B\in \mathcal B(E)$. Suppose that, for some $x\in E$, $$ P_x(\exists t\ge0 \text{ such that } X_t\in B)=1. $$ My question: Does there exist a stopping ...
1
vote
0answers
24 views

Renewal Process from Markov Chain

GIVEN: $X_0,X_1,...$ irreducible, recurrent Markov chain with transition matrix $P$ Starting state $X_0=x$ $g(m)=P\{X_m=y\}$ for some fixed state $y$ I know that the renewal process is $g(m)=b(m)+\...
0
votes
1answer
20 views

How can I compute the Limiting Distribution in the following problem?

Consider the transition matrix $ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $ for general $2$-state Markov Chain $(0 \le p, q\le 1)$. Find the limiting distribution ...
0
votes
0answers
30 views

Transition probability matrix, weather changes

I am trying to solve the following problem: and the probability that there will be three sunny days in a row specifically. I know that the answer is $1/5$ but I am trying to figure out how to get ...
1
vote
0answers
39 views

I'm trying to understand a weak convergence result for Feller processes in Ethier and Kurtz

Let $E$ be a locally compact separable metric space, $D([0,\infty),E)$ denote the Skorohod space, $C_0(E)$ denote the space of continuous functions $E\to\mathbb R$ vanishing at infinity and $E^\ast:=E\...
0
votes
0answers
58 views
+50

If $X^n$ is a càdlàg process in a locally compact space $E$, show that $\{X^n\}$ is relatively compact as processes in the compactification of $E$

Let $E$ be a locally compact separable$^1$ metric space, $D([0,\infty),E)$ denote the Skorohod space, $C_0(E)$ denote the space of continuous functions $E\to\mathbb R$ vanishing at infinity and $E^\...
1
vote
1answer
32 views

Càdlàg Feller process is quasi-left-continuous

I've been working in Chung's "Lectures from Markov Processes to Brownian Motion", and I got stuck at Exercise 1 from 2.4. The objective of the problem is to give a short proof of the quasi-left-...
1
vote
0answers
10 views

Expected Growth of General Pure Birth Process

Assume we have a non-explosive Markov Chain $(X_t)_t$ in continuous time with Q matrix $$ Q(k,l) = \begin{cases} \lambda_k &\text{if } l=k+1\\ -\lambda_k &\text{if } l=k\\ 0 &\text{...
1
vote
0answers
23 views

Reference request: Markov chain

I am in a situation to know whether a discrete time Markov chain evolving on Banach space $\mathbb R^n$ whose evolution (of states) equation we know and then I am interested to project( Hopefully an ...
1
vote
1answer
35 views

Equivalent Definitions of the Markov Property

Assume we have a stochastic process $\{X_n\}_\mathbb{N}$ defined on some underlying probability space that takes values in another measurable space. One of the many definitions that I have seen of ...
0
votes
0answers
18 views

transition function $P_t(k,\ell)$ for a Markov-process with $p(m,m+1)=1$.

Consider the $Q$-matrix on $\mathbb{N}$ defined by $q(m,m+1)=-q(m,m)=\rho m$ and $q(m,m')=0$ if $|m-m'|>1$, where $\rho$ is a constant. Letting $(P_t)$ be the transition function of this process, ...
2
votes
0answers
87 views

Strong markov property of a transformation of the Brownian motion

Let $(B_t)$ be a standard Brownian motion and consider $(X_t)$ defined as: $$X_t=e^{-t}B_{e^{2t}}$$ I've proved that this process is markov, however I can't prove that is strong markov. I know that ...
0
votes
0answers
22 views

Interrogating the results of a Markov-chain simulation - Help and feedback highly appreciated

I have built a Markov chain which simulates the daily routine of German residents (activity patterns). Each simulation day is divided into 144-time steps and the person can carry out one of fourteen ...
1
vote
0answers
21 views

Why $\mathbb P\{X_v\in A\mid \mathcal F_s\}=\int P_{s,t}(X_s,dy)P_{t,v}(y,A)$?

Let $X$ a stochastic process and $\mathcal F_s=\sigma (X_u\mid u\leq t)$. In the book "Continuous Martingale and Brownian motion" (third edition) of Yor and Revuz, page 80 : Let $$P_{s,t}(X_s,A)=\...
1
vote
1answer
23 views

Showing a process is a markov process

Let $P$ be a stochastic $N \times N$ matrix. We consider two independent copies of a Markov process on $\{1,\ldots,N\}$ with transition probabilities given by $P$. Both are assumed to start ...
1
vote
1answer
31 views

An application of Markov property

I'm a new learner on Markov chain, and I was confused about a small point when I use the Markov property to solve exercise 6.3.1 in Durrett's PTE(2010): Let $A\in\sigma(X_0, \cdots, X_n)$ and $B\...
1
vote
0answers
35 views

Conclude convergence in probability from uniform convergence on a set of limiting probability 1

Let $\kappa_d$ be a Markov kernel on $(\mathbb R^d,\mathcal B(\mathbb R^d))$ for $d\in\mathbb N$ $f_d:\mathbb R^d\times\mathbb R^d\to\mathbb R$ be Borel measurable for $d\in\mathbb N$ $B_d\in\mathcal ...
0
votes
0answers
7 views

Autocovariance and smoothness

I have just started to learn Time series from Shumway & Stoffer, 4th. My question is about the autocovariance. On page 25 there is something that I can’t understand it’s reason. Very smooth ...
0
votes
0answers
11 views

Kernel of a markov process

Could anyone just explain to me what does it mean by mathematically, the kernel $P_n(x, dy)$ is the law of $X_n$ here in the page $46$. https://statweb.stanford.edu/~cgates/PERSI/papers/iterate.pdf ...
2
votes
2answers
64 views

If $X,Y$ are random variables and $Y\sim\mathcal N(x,\sigma^2)$ if $X=x$, are we able to conclude $Y-X\sim\mathcal N(0,\sigma^2)$?

Let $\sigma>0$ and $$Q(x,\;\cdot\;):=\mathcal N(x,\sigma^2)\;\;\;\text{for }x\in\mathbb R.$$ Note that $Q$ is a Markov kernel on $(\mathbb R,\mathcal B(\mathbb R))$. Now, let $X,Y$ be real-valued ...
0
votes
1answer
28 views

Brownian motion remains nonnegative for some interval with length $1$ almost surely

Let $B_t$ be a continuous Brownian motion. I'm having a really difficult time to prove that the Brownian motion stays nonnegative for some interval with length $1$ almost surely. The reason for this ...
0
votes
0answers
12 views

Prove that $\ \lim_{m \to \infty} \mathbb{P}(T_0 < \infty|X_0 = m) = 0 \ $ where $T_0$ is the waiting time to reach state $0$.

Let $(X_n)_{n \geq0}$ be a Markov Chain on {$0,1,...$} with transition probabilities given by: $P_{0,0}=1, \ \ P_{i,i+1}= p_i, \ \ P_{i,i-1}= q_i, \ $ for all $i \geq 1$, and $P_{i,j}=0 \ $ otherwise. ...
0
votes
1answer
23 views

Decay rate of a birth-death Markov chain and relationship with the support of the orthogonalizing probability

I am studying this Article by van Doorn on the existence of quasi-stationary distribution for a birth-death process with killing. He defines the decay rate of a birth-death process with killing as $$...
0
votes
0answers
17 views

Why are the transition probabilities for an $M/M/1$ queue $\lambda$ and $\mu$?

The transition state diagram for an $M/M/1$ queue is shown as The only explanation I have for the transition probabilites is that arrivals occur at rate $\lambda$ according to a Poisson process and ...
4
votes
1answer
168 views

Markov chains. If $X_0=0$ then the probability that $X_n\ge1$ for all $n\ge 1$ is $\frac{6}{\pi^2}$

Let $(X_n)_{n\ge0}$ be a markov chain on $\{0,1,...\}$ with transition probabilities given by $p_{01}=1,p_{i,i+1}+p_{i,i-1}=1, p_{i,i+1}=\Big(\frac{i+1}{i}\Big)^2p_{i,i-1}, i\ge1$ I need to show ...
1
vote
2answers
59 views

Expected length of time until the mouse reaches the compost heap.

I'm assuming it's a Markov Chain question but I have no idea how to do this. Thanks in advance! A mouse lives in a mousehole with three exits. The first exit leads to a compost heap ...
0
votes
0answers
19 views

Strong Markov property and time-homogeneity

Let $(\Omega, \mathcal{A}, P, (X_n)_{n \in \mathbb{N}})$ be a discrete-time stochastic process on a state space $S$ (which we assume to be finite and discrete for simplicity). Denote by $\mathcal{F}_n ...
0
votes
0answers
67 views

Galton-Watson process with geometric offspring distribution and finite expected depth

Consider the Galton-Watson tree with offspring distribution $X$ given by $P(X=k) = (1-p)^kp$. Let $D$ be the depth of the process. My question is how to calculate for which values of $p$ we have $\...
0
votes
0answers
13 views

What would be the value of $X_1$ in case of a Limiting Distribution?

For the two-state chain with $$P = \begin{bmatrix} 1-p & p\\ q & 1-q \end{bmatrix}$$ the limiting distribution can be shown to be $$\lambda = (\frac{q}{p+q}, \frac{p}{p+q})...
1
vote
1answer
22 views

What does this Markov Chain notation mean?

So, I don't know the meaning of much of the notation present in the following slide. I know basically what a Discrete Markov Chain and a Transitional Probability Matrix are, but I'd like to what ...
0
votes
0answers
8 views

Can a continuous time Markov processes have transition densities only at certain times?

Suppose that $X=(X_t)_{t\ge 0}$ is a Markov process on some state space $E\subset\Bbb R^n$ with transition semi-group $(P_t)_{t\ge 0}$. It is often nice to know, whether the transition measures have ...
0
votes
0answers
13 views

Function of transient markov processes

Suppose that $X_t$ and $Z_t$ are Markov processes, which are $\mathbb{P}\otimes m$-a.e. (here $m$ is the Lebesgue measure on $\mathbb{R}$) not equal. If $X_t$ and $Z_t$ are transient, it seems right ...
1
vote
1answer
40 views

What's going on with this inequality?

Let $(\mathsf{X}, \mathcal{X})$ be a measurable space, $(\mathcal{F}_b(\mathsf{X}, \mathcal{X}), \Vert \cdot \Vert_{\infty})$ be the space of all bounded measurable functions on this space equipped ...
1
vote
1answer
23 views

Markov Jump Process - Transition Probabilities

For the transition probability of a Markov jump process from state $x$ to $y$ in a small time interval $\Delta t$, what is the meaning of $o(\Delta t)$? $\alpha$ is a constant $\in[0,1)$.
0
votes
0answers
16 views

Question Regarding Canonical Representation of Markov Networks

Referring to the Markov Random Field Wikipedia page, there is essentially nothing stopping them from writing a Markov network as an exponential family. Now let's say you have a simple Markov network ...
0
votes
1answer
21 views

Confused with the definition of hitting time (Markov chains)

Let $\emptyset \neq A \subseteq \mathcal{S}$ on a state space $\mathcal{S}$ of a Markov chain $\{X_n\}_{n=1}^\infty$. We define $$T_A := \inf\{n \geq 1 \mid X_n \in A\}$$ with $\inf \emptyset = +\...
1
vote
0answers
32 views

Generation theorem for Feller semigroups

Let $E$ be a locally compact Hausdorff space. I want to show that a linear operator $(\mathcal D(A),A)$ on $C_0(E)$$^1$ is closable and the closure $(\mathcal D(\overline A),\overline A)$ is the ...
1
vote
0answers
65 views
+50

If $X^n$ is a Feller process, $τ_n$ is a stopping time and $h_n\to0$, why can we conclude $|X^n_{τ_n}-X^n_{τ_n+h_n}|→0$ from $|X^n_0-X^n_{h_n}|→0$?

Let $E$ be a compact complete separable metric space and $X^n,X$ be $E$-valued càdlàg strong Markov processes on a probability space $(\Omega,\mathcal A,\operatorname P)$. Question 1: Let $\tau_n$ ...
2
votes
0answers
58 views

If $\kappa$ is a contractive operator on $C_0(ℝ)$, is $λ(κ-\text{id})$ the generator of a Feller semigroup?

Let $E$ be a locally compact separable metric space, $$C_0(E):=\left\{f\in C(E):\left\{|f|\ge\varepsilon\right\}\text{ is compact for all }\varepsilon>0\right\},$$ $\kappa$ be a Markov kernel on $(...
0
votes
0answers
31 views

Markov Process Expectation and Variance

Consider Markov process, where a person is searching for some particular piece of information among the list of all kinds of data. The first time the person studies some new piece of information, the ...
0
votes
1answer
35 views

Conditional expectation of integral of Ornstein-Uhlenbeck process

Given that $X(t)$ is an Ornstein-Uhlenbeck process with $X(0) = x_0$, which is a Markov process, but not a Martingale, how could I go forward if I would like to calculate $E[\int_0^T X(s)ds | \...
0
votes
1answer
25 views

What would be the value of $P$ in this example? (2)

A Markov Chain $(X_n)_n$ has the following transition matrix: $$P = \begin{bmatrix} 0.1 & 0.3 & 0.6\\ 0 & 0.4 & 0.6\\ 0.3&0.2&0.5 \end{bmatrix}$$ with initial ...
0
votes
0answers
8 views

mean and variance in backlog process

We have a discrete process $(N_k)_{k\in\mathbb{N}}$ defined as follows. At each time $k$ an amount of Poisson$(\nu)$ packets come in, but there is also a probability $1/e$ that exactly $1$ packet ...
5
votes
2answers
60 views

Definition of a Markov process: What does $\mathbb P_x\{X_u\in B\mid \mathcal F_t\}=p(u-t,X_t,\Gamma)$ mean?

I am reading the book Random perturbation of dynamical sustem of Freidlin and Wantzell (2nd edition). On page 20, they define a Markov process as follow: Let $(\Omega ,\mathcal F,\mathbb P)$ a ...
0
votes
1answer
49 views

What would be the value of $P$ in this example?

A Markov Chain $(X_n)_n$ has the following transition matrix: $$P = \begin{bmatrix} 0.1 & 0.3 & 0.6\\ 0 & 0.4 & 0.6\\ 0.3&0.2&0.5 \end{bmatrix}$$ with initial ...
0
votes
0answers
7 views

Markov chains and identifying patterns in phone numbers using R

I have a very long list of phone numbers. There are about 500 unique phone numbers and several thousand calls in total. They are all in time order. I want to look for patterns in this. My initial ...
0
votes
1answer
27 views

Expected number of steps before leaving a ball

Consider an infinite undirected graph $G$, like for example $\mathbb{Z}^d$ with edges connecting nearest neighbours sites. Let $X(t)$ be a simple random walk starting from the origin, $o$, define $ ...