Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

10
votes
0answers
309 views

Uniqueness of an infinite system of linear ODEs

How to prove that $\dot{x}=ax,\space x(0)=1$ has a unique solution if $a,x$ are infinite dimensional matrices? More specifically, let $Q$ be a bounded infinitesimal generator, i.e. $Q=(q_{i,j})_{i,j\...
9
votes
0answers
275 views

Limit distributions for Markov chains $X\to\sqrt{U+X}$

This question spawned from a recent, very interesting problem. Let $\varphi=\frac{1+\sqrt{5}}{2}$ and $T$ denote the map on the space of continuous probability density functions supported over $\...
6
votes
0answers
250 views

Poisson: Conditional Probability on Pizza order

I am not sure about my answer. In particular, part b of the following question. Pizza orders arrive according to a Poisson process of rate 20 per hour. Orders are independently for a vegetarian ...
5
votes
0answers
372 views

Proof of strong Markov Property of double sided Levy Process

Let $(X_t)_{t\in\mathbb{R}}$ be a Levy Process, i.e. $X_0 = 0$ a.s., $X$ has independent and stationary increments, and almost all paths $t\mapsto X_t(\omega)$ are right continuous with left hand ...
5
votes
0answers
141 views

A Continuous-Time Markov Process Taking All Possible Values

Let $\mathbb{N}$ be the set of positive integers. For each $n \in \mathbb{N}$, let $X^{(n)}=\{ X^{(n)}(t): t \geq 0 \}$ be a Markov chain with state-space the two point set $\{0,1\}$ and $Q$-matrix \...
4
votes
0answers
348 views

Why are coordinate processes introduced?

I don't fully understand what a coordinate process is and why they are used. My lecture notes say the following: For an arbitrary measurable space $(S,\mathcal{S})$ we denote by $S^{[0,\infty)}$ ...
4
votes
0answers
85 views

Sum of exponent of jump times

I have the following question. I have a random variabele that is the sum of the exponent of the jumping times $T_i$ of a Poisson process $N(\cdot)$ with parameter $\lambda$. Say: $$B(t) = \sum_{i=...
4
votes
0answers
205 views

Intuition - strong markov property vs. stationary

I try to gain an intuition what a difference between Markov property, strong Markov property and stationarity of a random process is. I'm physicist with no strong background in stochastic theory. I ...
4
votes
0answers
66 views

Question about a type of continuous state Markov-process.

EDIT: Solved! It turns out that if the function is continuous and various regularity conditions hold then the statement is true. This has been established in the 'stochastic approximation' literature, ...
4
votes
0answers
126 views

Blackwell's example in Markov process theory and Kolmogorov's extension theorem

I'm reading Continuous Time Markov Processes: An Introduction by Thomas M. Liggett. Chapter 2.4 is devoted to Blackwell's example. Let $E=\left\{0,1\right\}$, $\mathcal E:=2^E$ and $X$ be the (...
4
votes
0answers
250 views

Limit of decreasing sequences of markov time (stopping time) is markov time?

Let $(\Omega, \mathcal{F}, \{\mathcal{F}_t\}_{t \geqslant 0}, \mathbb{P})$ be a filtered probability space and let $\tau_n \geqslant \tau_{n+1}$ be a markov time (stopping time) with respect to ...
4
votes
0answers
242 views

Interpretation of potential kernels for Markov processes

One can associate a strongly continuous contraction semi group (SCCSG) to a Markov process with state space $S$ through its transition function, say $P_t$. Now one can interpret $P_t$ as a linear ...
4
votes
0answers
175 views

Conditional probability and integrating out part of a random walk

Suppose that I have a random walk process defined by $\alpha_{t+1}$ ~ N$(\alpha_t, \omega^2)$. Given $\alpha_t$ and $\alpha_{t+2}$, I understand why the conditional formula for $(\alpha_{t+1}|\alpha_{...
4
votes
0answers
418 views

estimation of transition probabilities from aggregate data

Please, O mathematicians, help me understand the approach to the problem of estimating transition probabilities given only aggregate data in Kalbfleisch & Lawless' 1984 paper "Least-Squares ...
4
votes
0answers
532 views

Is every killed Markov process still a Markov process?

Suppose we've got $X=(X(t))_{t\geq 0}$. $X$ is a strong Markov process with respect to filtration $\mathcal{F}_t$, taking values in some subset of $\mathbb{R}$. We take $\tau$ - a stopping time w.r.t $...
3
votes
0answers
63 views

Show that if $(κ_t)_{t≥0}$ is the transition semigroup of a strong solution to an SDE, $t↦(κ_tf)(x)$ is continuous for all $x$ and suitable $f:ℝ→ℝ$

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $b,\sigma:\mathbb R\to\mathbb R$ be Lipschitz continuous and $$Lf:=bf'+\frac12\sigma^2f''\;\;\;\text{for }f\in C^2(\mathbb R)$$ $W$ be ...
3
votes
0answers
23 views

Question about integral notation in a Markov process + how to evaluate said integral

I'm reading Chapter 11 of Puterman's book on Markov Decision Processes (in particular, about continuous-time Markov processes). There's a lot of notation involved, but I've tried to distill the ...
3
votes
0answers
80 views

How to find probability transition matrix for continuous time markov chain?

In Grimmet and Stirzaker, on page 258 it explains how to find transition probabilities, given a generator matrix: (a) nothing happens during $(t,t+h)$ with probability $1+g_{ii}*h+o(h)$ (b) ...
3
votes
0answers
42 views

Markov Property fos Ising type Models

We are interested in proving the Markov property for the long range Ising type model in $\mathbb{Z}^d$. Setting: Define $\Omega = \{-1, +1\}^{\mathbb{Z}^d}$ the space of all possible configurations ...
3
votes
0answers
123 views

Will simple random walk on $n$-cycle converges to Brownian motion on $S^1$?

I know that, by Donsker's theorem, simple random walk on $\mathbb{Z}$ will converge to Brownian motion on $\mathbb{R}$. Here, simple random walk means that the Markov chain with probability from $n$ ...
3
votes
0answers
54 views

How to convert a detailed mathematical, statistical compuational process into mathematical and statistical equation?

I am working on a problem that allows haplotype phasing. I have developed this computation method (expressed in detail below) and have developed a python code (not shown here) to solve the issue. ...
3
votes
0answers
75 views

Stationary distribution for a Markov Chain on an uncountable space

Suppose $X_n$ is a Markov chain on an uncountable state space $\Omega$ (We can assume $\Omega=\mathbb{R}$), and suppose that $X_n$ is irreducible, aperiodic and Harris recurrent. What are the extra ...
3
votes
0answers
86 views

Hidden Markov Model: The current observation depends on the previous observation

This question is for the case of homogeneous discrete HMM's. In the regular HMM's, the probability of the current state depends only on the previous state, that is Pr(S_t|S_1,S_2,...,S_(t-1)) = Pr(...
3
votes
0answers
100 views

Find steady state of continuous time Markov chain

I have a Continuous Time Markov Chain with transition rate matrix given by the folowing recipe: Let $\lambda_1,\lambda_2 \geq 0$ and $\lambda_N := \lambda_1 + \lambda_2$. We denote $e_n$ for the row ...
3
votes
0answers
22 views

Conditional probability, a question concerning Kipnis and Cocozza paper from 1977

In the paper Existence de processus Markoviens pour des systèmes infinis de particules by Cocozza, C. and Kipnis, C. (Ann. lnst. H. Poincaré, Sect. B, 13, 239-257, 1977), one reads A transition ...
3
votes
0answers
62 views

Distribution on number of revisits in past $k$ steps of Markov chain

Consider a finite-state Markov chain with transition matrix $P$. The chain starts in a state chosen uniformly over all the states and runs indefinitely from there. We're going to examine only the $k ...
3
votes
0answers
296 views

How to handle Finite-state-machine with correlated inputs?

My system can be represented by the following state-diagram. where each arch represents Input/Output when a transition is made from one state to the other. The inputs to this FSM are correlated. ...
3
votes
0answers
273 views

This is a Markov Chain?

Consider two irreducible ergodic Markov chains with the same state space $\{0, 1, . . . , N\}$, with transition matrices $P$ and $Q$ and respective stationary distributions $\pi$ and $\rho$. We ...
3
votes
0answers
313 views

Direct proof of the existence of optimal memoryless deterministic policies in MDP

It is well known that (finite-state, finite-action, discrete time) MDPs admit an optimal policy that is memoryless and deterministic (sometimes called pure). The proof of this fact for discounting-...
3
votes
0answers
449 views

Markov process and filtration

I would like to restate the question. I'm reading Revuz/Yor's definition of Markov process (P81), they started from transition function, and define the $P_t f(x)$ as usual (let's only consider the ...
3
votes
0answers
624 views

Moment Generating Function for Brownian motion's exit of interval.

Let $B(t)$ be a standard BM. Consider the stopping time $T = \inf\{ t > 0: |B(t)| = a\},$ the usual first exit time of the interval $(-a, a).$ We can see that $\mathbb{E} e^{tT} < \infty$ for $...
3
votes
0answers
86 views

Correct steps in rewriting expectation to a probability

My knowledge of measure theory and probability spaces is limited, so please keep it relatively simple. Let $\{X(t), ~ t \ge 0\}$ be a Markov process on the countable state space $\mathbb{N}_0$ with ...
3
votes
0answers
100 views

Discrete-Time Stochastic Calculus and Stopping Times: Resources

In my measure-theoretic probability course we covered what the professor called "discrete-time stochastic calculus". Essentially, it was a three part method for computing certain quantities such as ...
3
votes
0answers
144 views

Hidden Markov Model Transition Probability

I am doing my assignment and I am asked to derive transition probability of a HMM. There are Three states. H, E and T. They initially gave me the information as follow. E is followed by an H 40% of ...
3
votes
0answers
392 views

Generator of Wiener process and its running maximum

If we let $W$ be a standard linear Wiener process issued from zero and $M$ its running maximum $$ M_t := \sup \{ W_u: u \leq t \}, $$ then we could show that $(X,Y):=(M,M-W)$ is a Markov process on $\...
3
votes
0answers
407 views

Absorbed brownian motion is a Markov process

I have been asked to prove that the Brownian motion absorbed at the origin is a Markov process. Formally, let $B_t^x$ be a Brownian motion originating from $x>0$ and let $\tau^x_0 = \inf\{t>0 : ...
3
votes
0answers
116 views

Discrete Laplacian

I have the following question and I can't figure out how to do the proof. Could you give me some hints in both directions of the equivalence? Suppose $A$ is a bounded subset of $\mathbb{Z}^d$. Then $\...
3
votes
0answers
116 views

Simple random walk conditioning on non-return

Consider a simple symmetric random walk on $\mathbb{Z}$, $(S_t)_{t \geq 0}$, with $S_0=0$. Let $k$ and $j$ be two positive integers. Let $P_{k,j}$ be the probability that the walker hits the vertex $k$...
3
votes
0answers
451 views

References for time-inhomogeneous Markov jump processes?

In some central models in life insurance mathematics, the state of the insured is modeled using a continuous-time time-inhomogeneous Markov process with finitely many states. While many results for ...
3
votes
0answers
57 views

Exsistence and uniqueness of stationary density for Markov Chain

Suppose we're given a function $f:\mathbb{R}^2\to\mathbb{R}$. We define a Markov Chain $(X_n)$ by \begin{align} X_0&\sim f_X, \\ X_n&=f(X_{n-1},Y_{n-1}), \end{align} where $(Y_n)$ is a ...
3
votes
0answers
214 views

Generalization of Dobrushin's Ergodic Decomposition for continuous Markov Chains

Let $T$ be the shift transformation. Let $P$ be invariant for $T$ and also define a discrete state space Markov Chain. Let $C_{1},\ldots,C_{n}$ be the connected components of the Markov Chain. It ...
2
votes
0answers
19 views

Given a Markov process, are we able to construct another Markov process with the same transition semigroup but different inital law?

Let $E$ be a locally compact separable metric space $(T(t))_{t\ge0}$ be a strongly continuous contraction semigroups on $C_0(E)$ with generator $(\mathcal D(A),A)$ $(\Omega,\mathcal A,\operatorname P)...
2
votes
0answers
26 views
+50

Convergence of Feller processes implies convergence of the resolvent operators of their generators

Let $E$ be a locally compact separable metric space $(T_n(t))_{t\ge0}$ and $(T(t))_{t\ge0}$ be strongly continuous contraction semigroups on $C_0(E)$ with generator $(\mathcal D(A_n),A_n)$ and $(\...
2
votes
0answers
20 views

Strong Markov property and another stopping time

I'm trying to prove that given a regular continuous time Markov chain $X_t$ (pure jump process), its embedded chain given by $Y_n=X_{T_n}$ is a homogeneous Markov chain, where $T_n$ is the time of the ...
2
votes
0answers
21 views

Sufficient condition for the Markov property for a process in continuous time with values in a countable space

I would like to verify the folowing: Let $(X_s)_{s \in [0,\infty)}$ be a stochastic process in continuous time with values in a countable space $E$ and let $\mathcal F_s=\sigma(X_r: 0 \le r \le s)$ ...
2
votes
0answers
94 views

If $X^n$ is a càdlàg process in a locally compact space $E$, show that $\{X^n\}$ is relatively compact as processes in the compactification of $E$

Let $E$ be a locally compact separable$^1$ metric space, $D([0,\infty),E)$ denote the Skorohod space, $C_0(E)$ denote the space of continuous functions $E\to\mathbb R$ vanishing at infinity and $E^\...
2
votes
0answers
94 views

Strong markov property of a transformation of the Brownian motion

Let $(B_t)$ be a standard Brownian motion and consider $(X_t)$ defined as: $$X_t=e^{-t}B_{e^{2t}}$$ I've proved that this process is markov, however I can't prove that is strong markov. I know that ...
2
votes
0answers
90 views

If $X^n$ is a Feller process, $τ_n$ is a stopping time and $h_n\to0$, why can we conclude $|X^n_{τ_n}-X^n_{τ_n+h_n}|→0$ from $|X^n_0-X^n_{h_n}|→0$?

Let $E$ be a compact complete separable metric space and $X^n,X$ be $E$-valued càdlàg strong Markov processes on a probability space $(\Omega,\mathcal A,\operatorname P)$. Question 1: Let $\tau_n$ ...
2
votes
0answers
72 views

If $\kappa$ is a contractive operator on $C_0(ℝ)$, is $λ(κ-\text{id})$ the generator of a Feller semigroup?

Let $E$ be a locally compact separable metric space, $$C_0(E):=\left\{f\in C(E):\left\{|f|\ge\varepsilon\right\}\text{ is compact for all }\varepsilon>0\right\},$$ $\kappa$ be a Markov kernel on $(...
2
votes
0answers
30 views

Generalizations of the Reflection principle for Brownian motion

The reflection principle for Brownian motion roughly states that a Brownian motion reflected a stopping time is also a Brownian motion. More precisely, if $W$ is a Brownian motion and $T$ a stopping ...